General troubleshooting hints are included in the Aether OnRamp Guide, but some of the issues people report are due to the specifics of either their local environment or the hardware they are using. This page summarizes such issues and how they might be addressed.

Known Issues

1. AMP (specifically, ROC) does not cleanly uninstall/reinstall. The problem is being worked on, but until it is resolved, the only workaround is to do a fresh reinstall of Kubernetes.

2. When connecting multiple UEs to a physical gNB (e.g., five moto g 5G phones to MOSO CANOPY), it has been reported that only four connect at a time. You have to turn one off to connect a new one. This appears to be an issue with an old version of the gNB software that can be corrected with an update.gNBsim will sometimes report 4 of 5 UEs successfully connecting. This is likely a race condition that has been fixed in a development version of the webui image, and is not indicative of an OnRamp config/install problem. The following edit will work around the problem until a new version of SD-Core is released:

diff --git a/roles/core/templates/sdcore-5g-values.yaml b/roles/core/templates/sdcore-5g-values.yaml
index 07ac4f7..c8cb8c0 100644
--- a/roles/core/templates/sdcore-5g-values.yaml
+++ b/roles/core/templates/sdcore-5g-values.yaml
@@ -17,8 +17,8 @@ omec-control-plane:
   enable5G: true
   images:
     repository: "registry.opennetworking.org/docker.io/"
-    #tags:
-    #  amf: <amf image tag>
+    tags:
+      webui: gabearrobo/5gc-webui:0.0.1-dev
     # refer to above Helm Chart to add other NF images

Frequency Bands / UEs / PLMNs

The OnRamp guide recommends the MOSO CANOPY 5G INDOOR SMALL CELL (Model SCD5164), which is also known as the SERCOMM Bridgestone gNB (SCE5164-B78/-B48/-B77). The product data sheet implies support for multiple frequencies, including n78 and n48, but individual devices may be limited to just one (e.g., n48, as shown below).

Image RemovedImage Added

Not all UEs support all bands, and even when they do, there appear to be other limitations. For example, there is a limited set of UE’s that support the CBRS PLMN (315010). The following is the list of UEs that should work for n48, with the first being the only one confirmed to date:

...

With Pixel phones, it is best to use PLMN 00101; SERCOMM is seeing issues with 315010.

Programming SIMs

The recommended OYEITIMES MCR3516 SIM Card Writer is for 4G, but can also be used for 5G (albeit without the full set of service options). It requires running the SIM App on a Windows machine. After inserting a blank card, click the "Read Card" button at the top. This will fill many of the fields with their initial/default values. The following describes what fields you need to modify, resulting in the set of final values shown in the following screenshot. When you are done modifying the SIM, click the "Write Card" button to write your changes to the card.

Image RemovedImage Added

Many of the default settings are "good enough" to get started, but will eventually need to be changed if you plan to deploy a secure/production-ready network. For example, the green shaded boxes in the Common Parameters panel (e.g., PIN1, PUK1, PIN2, PUK2, ADM) are keys used to secure the SIM. The default settings will work, but will not protect the SIM from malicious actors.

...

An alternative approach (not yet verified to work with Aether) is to follow the procedure outlined by the srsRAN project, which includes: (1) an open source program (pysim) to program SIMs, and (2) a set of pre-configured (but reprogrammable) SIM cards from Sysmocom.

Configuring MOSO CANOPY 5G INDOOR SMALL CELL

We recommend the following recipe to get the MosoLab gNB running the first time. Most out-of-the-box settings are fine. If you've already changed some of them and want to start over, visit:

...

Verify that your configuration changes were applied once the radio comes back up.

Default GW

The above Gateway configuration assumes DHCP assigns an IP address to your gNB, but that the address is not likely to change, making it safe to treat it as static. The following elaborates on this.

...

You can verify that your server is successfully forwarding traffic by running the PING or TRACEROUTE diagnostic tool on the gNB.

MongoDB CrashLoopBackOff

MongoDB not successfully starting (getting stuck in CrashLoopBackOff state) may be due to an older CPU that does not support the AVX instruction required by MongoDB 5.0+. See the following post to the MongoDB Developer Community. If running on a VM (e.g., selecting host processor on Proxmox), keep in mind that you may be assigned a problematic CPU.

Source/Destination Checking on AWS

When running gNBsim and SD-Core on separate servers in AWS, if summary.log reports "UEs Passed: 0, UEs Failed: 5", then it might be that AWS settings are interfering with user plane traffic. Digging deeper, when gnbsim.log shows PDU session establishment is successful but data packets are not transferred between gNBsim and the SD-Core, then User Plane routing is the likely culprit. If you have verified the routes are installed correctly, then it may be the case that AWS has source/destination checking enabled. To disable source/destination checking:

1. Open the Amazon EC2 console.
2. In the navigation pane, choose Instances.
3. Select the instance (you will need to repeat for both the Ansible master and worker nodes).
4. Choose Actions, Networking, Change source/destination check.
5. For source/destination checking, select Stop.
6. Click Save.

Network Service on Ubuntu

Aether assumes Ubuntu Server, in part because Ubuntu Desktop typically disables systemd.networkd and does not have netplan installed. This means Ansible is unable to find the .network file, which results in a failed task in the Ansible playbook for the route role when installing 5gc. If Ubuntu Desktop is your only option, one workaround is to install netplan with

...

Bottomline: You need to identify the least intrusive solution, which may be challenging if you end up mixing management directives between systemd.networkd and the Network Manager. The best approach is to start with Ubuntu Server in the first place.

Multiple Instantiations of Aether

As noted in the Troubleshooting hint in the QuickStart Section of the OnRamp Guide:

"Another possibility is that you have multiple SD-Cores running in the same broadcast domain. This causes ARP to behave in unexpected ways, which interferes with OnRamp’s ability to establish a route to the UPF pod." 

This is because each OnRamp-deployed instance of Aether (whether it runs in a single server, as is the case with QuickStart, or is scaled across multiple servers) has a fixed pair of IP subnets allocated to the Core's UPF: 192.168.252.0/24 and 192.168.250.0/24, both of which are associated with the server's physical NIC. Making these prefixes configurable on a per-deployment basis is on the Roadmap for OnRamp, but if you want to deploy multiple instances of Aether today, the easiest workaround is to manually edit those prefixes throughout each clone of OnRamp you plan to deploy). For example, you might change the subnets to 192.169.252.0/24 and 192.169.250.0/24 for the second instance. Be aware that these subnets are defined at multiple places throughout each copy of OnRamp. (The plan is to define the per-Aether prefixes just once in vars/main.yml and use templating to set all the uses.)