For an archive of meeting notes pre-dating the Aether Project moving from ONF to LF, click here.
...
- Component = 'Release Engineering'
- Product = Aether
3-
...
18-25 (Topics for Next Meeting)
- ...
3-
...
11-25 (Today's Agenda)
- Infrastructure
- GitHub Actions
- Pushed an initial version last night
- https://github.com/onosproject/.github/pull/7 AWS Images
- New 22.04 works
- New 24.04 fails in Docker install
- Apache 2.0 uses more restrictive language than MIT, but they are roughly equivalent
- Document in README (probably sufficient at the "GitHub organization README" level)
- Currently addressing issues related to new golang dependency versions
- Newer golang dependencies require semver format for the go module (i.e., go 1.23.0 instead of go 1.23)
- Significant progress on 3GPP release upgrade (release 18): All NFs are now with release 18 for the SBI
- Several issues to address such as callbacks, replace some structs that were removed in newer 3GPP release, among others
- Open Source Near-RT RIC Comparison Report (by Rimedo Labs) - see today's blog post
- Planning a weekly blog highlighting SD-RAN features
- Culminates in a webinar to bring it all together (May 27th) - register for webinar
- SD-RAN v1.5 Techinar Video
- srsRAN has been tested with other RICs, but...
- SD-RAN sends an "E2 Connection Update" message that srsRAN does not respond to
- SD-RAN may need to be robust when there is no reply
- Blueprint for "Multi-Hop Access" network released
- A student working on service monitoring; just opened a PR
- General idea: eliminate per-NF monitoring; always go through metric_func
3-4-25
- Infrastructure
- GitHub Actions
- Pushed an initial version last night
- https://github.com/onosproject/.github/pull/7
- AWS Images
- New 22.04 works
- New 24.04 fails in Docker install
- See aether-packer repo: https://github.com/opennetworkinglab/aether-packer
- See PR for aether-jenkins: https://github.com/opennetworkinglab/aether-jenkins/pull/16
- GitHub Actions
- SD-Core
- New Helm Charts were released (version 2.2.2)
- Fixes (avoids) 100% CPU utilization
- 3GPP Release upgrade (release 18)
- SMF: Addressing issue with multipart response message
- NRF: Addressing missing callback
- How to deal with multiple licenses; e.g., MIT & Apache (ask for LF advise)
- New Helm Charts were released (version 2.2.2)
- SD-RAN
- SD-RAN v1.5 Techinar is tomorrow, March 5th at 8:00am Pacific. Register to attend
- OnRamp
- Reviewing latest PR with 2.2.2 upgrade.
- Discussion about scope: Ease learning curve or industrial strength ?deployment robustness
- Testing Infrastructure assumes OnRamp itself is relatively static
- ARA
- Discussion with Serdar Vural from Canonical about Charmed Aether SD-Core
2-25-25
- Infrastructure
- GitHub Actions
- Eric Update: ROC release and reusable workflows: After speaking with one of my colleagues last week who does a lot of work on our reusable workflows, I rearchitected the helm release to have most functionality outside of the workflow, where pretty much any action that needs to be taken is called from a small, discreet "micro-action". The result is workflows that are far cleaner, and having each bit of work in its own action allows for much clearer documentation. Even if it's still at a WIP point, I'll push my code later so folks can get a look at it and provide feedback.
- AWS Images
- will create a new
aether-packer
repo for VM spec - plan is to support 22.04 and 24.04; likely to deprecate 20.04
- Eric Update: I've got code-based, reproduceable builds of Ubuntu images for Jenkins up and running. I'm currently testing the produced AMIs with this job: https://jenkins.aetherproject.org/view/all/job/Aether_OnRamp_Manual-test/. It looks like the most recent attempt failed due to a missing key. That may actually be something that needs to be changed in the job, since the best practice would be to have the key stored on the Jenkins server rather than embedded in the image.
- will create a new
- GitHub Actions
- SD-Core
- New Helm Charts are released (version 2.2.1)
- Remove cert-manager and use of self-signed secrets
- Issue with "stagnant" dependencies
- golang.org/x/crypto from 0.33.0 to 0.35.0: https://github.com/omec-project/webconsole/pull/313
- https://pkg.go.dev/gopkg.in/yaml.v2: Published: Nov 17, 2020
- Discussion about MIT vs Apache license; will ask for LF's input (llp action item)
- Webconsole and 4G support
- Do we need to be backward compatible with 4G? → We have decided to not support 4G, but looking for Ajay's input
- New Helm Charts are released (version 2.2.1)
- SD-RAN
- Webinar (announcing v1.5) planned for March 5th.
- Will finalize v1.6 planning after the webinar
- Hardware documentation is being updated
- Team won best paper award at a Korean conference (will update Research page)
- OnRamp
- srsRAN blueprint has regressed
- Currently, OnRamp only supports physical/external gNBs connected to the same subnet as 5GC host machine
- This will cause issues on how to reach the UPF's N3 interface: This issue is present in ARA and other deployments
- PRs https://github.com/opennetworkinglab/aether-5gc/pull/36, https://github.com/opennetworkinglab/aether-srsran/pull/10 address this limitation.
- Need to downgrade K8s (RKE) by one minor version due to issues finding compatible Helm Charts for the monitoring component
- There is a "NetPlan" issue (see Slack)
- ARA
- No update.
- ...
...
- LF "Core" Workshop
- Interest in gNBsim as a core-neutral testing tool → Easy adoption
- Interest in shared tooling for generating 3GPP APIs
- Interest in shared support for physical UEs (e.g. burning SIM cards)
- Comparison Cheatsheet → Ease of getting started (Aether strength)
- See notes: https://lf-networking.atlassian.net/wiki/spaces/LN/pages/332726273/Open+Source+Packet+Core+Workshop
- Infrastructure
- GitHub Actions
- Issues with golanci-lint: Had to close dependabot-opened PRs
- AWS Image Instances
- Enable private security reports
- GitHub Actions
- SD-RAN
- SD-RAN 1.5 webinar planned for March 5th, 8am Pacific
- Added the comparative analysis to Wiki (will mention on Slack too)
- Started demo recording for webinar.
- Started investigating how we can merge @Gabriel Arrobo’s PRs for 5G RSM use-case.
- Started 1.6 planning - especially for the new RAN-Simulator and API/go version upgrade.
- SD-Core
- Fixed AMF issue; Helm chart update pending (multiple fixes to include)
- Looking at adding a CERT manager sub-chart
- Review security-related issues/PRs opened PSU team (Gabriel)
- Need to prune https://docs.sd-core.aetherproject.org of obsolete/deprecated info
- e.g., references to NG40, 4G support, etc.
- Continue having progress with release upgrade:
- Currently testing changes with SMF
- Still addressing some "compatibility" issues
- OnRamp
- Updated to RKE v1.25.16+rke2r2 and Helm v3.17.0 ( Thanks Gabriel! )
- Remove the need of installing kubectl from source
- srsRAN blueprint seems to have regressed
- Started email exchange with Jordan & Sean (Atomix & ROC)
- Due to loading CRDs; need to disaggregate
- Will likely deprecate support for 20.04 once we are fully supporting 24.04. → to discuss
- Updated to RKE v1.25.16+rke2r2 and Helm v3.17.0 ( Thanks Gabriel! )
- ARA Deployments
- SD-RAN
- Working on the support of K8s
- SD-RAN
- ...
...
- Infrastructure
- GitHub Actions
- AMIs for Jenkins
- Discuss recent reports
- RIC Evaluation: https://drive.google.com/file/d/13Lyqf0l91FEi-9HB7E99-u_aIbJVmYB7/view?usp=drive_link
- What can the Aether project do better to support its nRT-RIC → dependabot (security alerts)
- Sufficient developer community / user community
- Table further discussion until Woojoong is available
- Security Report (with respect to SD-CORE): https://nathanielbennett.com/publications/ransacked.pdf
- Almost entirely focused on 4G → We do not maintain 4G, and need to be clear about this.
- Action Items:
- Reflect this position on WIki, docs, website
- Invite the researcher to a future TST call
- Enable private security reports: https://docs.github.com/en/code-security/security-advisories/working-with-repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository (done for some repos; need to make sure on all active repos)
- Done for all 5G-related repos of SD-Core.
- RIC Evaluation: https://drive.google.com/file/d/13Lyqf0l91FEi-9HB7E99-u_aIbJVmYB7/view?usp=drive_link
- OnRamp
- Why ROC's Atomix is deployed in the core Kubernetes namespace (kube-system)? [Gabriel]
...
- Infrastructure
- GitHub Actions
- Generate AMI instances
- SD-Core
- Webconsole changes by Canonical
- Extending API to support adding UPFs/Slices
- Should not have impact on legacy API
- Includes swagger support
- Article on security flaws in open source core:
- https://www.securityweek.com/lte-5g-vulnerabilities-could-cut-entire-cities-from-cellular-connectivity/
- PDF of academic report: https://nathanielbennett.com/publications/ransacked.pdf
- To revisit next week
- Webconsole changes by Canonical
- SD-RAN
- Draft evaluation report: https://drive.google.com/file/d/13Lyqf0l91FEi-9HB7E99-u_aIbJVmYB7/view?usp=drive_link
- Let's review and plan to discuss next week
- anything we disagree with? If so - raise it now before this is finalized
- what do we do this this now? Anything actionable?
- Press/PR opportunity? Can we tease out some positives to publicize?
- Reviewing an announcement of v1.5
- Working on webinar; to include a demo
- OnRamp
- Jenkins now stable enough to spot regressions
- Multi-UDP Test has been failing since Jan 2nd
- Possibly related to SD-Core chart upgrade from 2.0.0 to 2.1.0?
- See for example https://jenkins.aetherproject.org/job/AetherOnRamp_UPF_22.04/102/
- ARA Deployments
- SD-RAN
- Preparing SD-RAN ARA blueprint
- Working on K8S support on ARA portal
- Take note of RKE2 dependency
- SD-Core
- SD-Core + Ericsson RAN blueprint review
- SD-RAN
- Other Topics
- ...
...
- Monogoto Introduction (https://monogoto.io/)
- Infrastructure
- Jenkins updated to recent LTS
- Aether Registry includes a proxy for Docker Hub
- Also a daily script that copies latest images from Docker Hub to Registry
- SD-Core
- Enforcing MongoDB ReplicatSet or Sharded Cluster for transaction support (Ghislain)
- Standalone mode doesn't support transactions
- Should just work
- Support AF_PACKET mode in helm charts to enable E2E tests in GH actions (Ghislain)
- Working towards e2e tests for PRs
- May run into "GH runners" limitations and need Jenkins involvement
- If this proves to be the case, existing Jenkins "Quickstart" job is close (e2e gnbsim+sdcore)
- would need to modify to work on candidate images
- Review, test, and merge ~20 PRs opened by Dependabot (dependencies)
- Adjusting log level to reduce performance penalty
- Enforcing MongoDB ReplicatSet or Sharded Cluster for transaction support (Ghislain)
- SD-RAN
- Reviewing release notes
- Will announce release
- Then will work on webinar
- Looking to August for 1.6 release
- OnRamp
- SD-Core playbooks updated to use registry.aetherproject.org/proxy/
- SD-RAN and ROC are still pending
- Connect Pixel7a phone with srsRAN blueprint (USRP X310, band 78)
- ARA
- SD-RAN
- Checking O-RAN Alliance specifications on the E2 Service Model and SD-RAN code.
- Exploring the potential development of xApps for spectrum sharing and network slicing
- SD-Core
- UE barring issue troubleshooting with Ajay on Slack. Need to provide more logs.
- SD-Core + Ericsson RAN blueprint refinement
- SD-RAN
- ...
...
- Infrastructure
- DNS names cleaned up
- docs.sd-ran.aetherproject.org, docs.sd-ran.org, and docs.sd-core.aetherproject.org work as expected
We don't have control over sd-core.org
sd-ran.org and sd-ran.aetherproject.org redirect to aetherproject.org
- docs.sd-ran.aetherproject.org, docs.sd-ran.org, and docs.sd-core.aetherproject.org work as expected
- Rate-Limits
- For information only – OMECPROJECT Docker Pull Counts
- No progress on plan
- Certificates
- Need authorization for Slack channel update to #aether-tst – pending
- DNS Sites
- Will be moving *.opennetworking.org to *.aetherproject or *.volthaproject domains as pertinent
- May require additional community and development education.
- DNS names cleaned up
...
- OnRamp
- PR related to k8s configuration pending - meeting with Gabriel, Suresh and Larry to sync up on this
- Infrastructure
- Confluence
- The wiki.aetherproject.org site is now read-only as we migrate the contents to the Atlassian Cloud
- The old site will remain available in read-only mode at wiki.archive.aetherproject.org for at least one month.
- Confluence
- ROC
- No progress on updating Wiki - Sean to continue on this
8-20-24
- SD-RAN
- Working on v1.5 release
- Working on documentation and release notes
- To include OAI 5G RAN
- includes "ONF" E2 agent (not OAI's)
- implies need to rebuild OAI images (since E2 agent is statically included)
- Not include update to golang
- Expect announce 1st week of September
- E2SM → Will add O-RAN RC Service Model
- Backward compatible with the old ONF Service Model
- Working on v1.5 release
- Infrastructure
- Jira
- Production site available at https://jira.aetherproject.org
- Still working out Anonymous creation of issues
- Permission Question:
- Currently four levels of access
- Anonymous (users without Atlassian accounts or not logged into them) can Browse and Read existing Issues but not modify anything
- Any Logged in user can basically change almost any field in an issue, and invite people to join the site and become a user
- When invited, specific domains are automatically approved when invitee makes the request
- Basically, any TSC user's domain (that is NOT public (ie, not gmail, yahoo, aol, ...))] is currently automatically approved.
- aether-jira-admin users can update group and user permissions
- admin can do anything.
- QUESTION: Any need now to restrict (beyond being logged in as an approved Aether Jira user) things like:
- Transitioning a ticket between states (including closing)
- Mucking with assignee
- editting fields beyond comments
- → Taking a wait-and-see approach
- Currently four levels of access
- Known TODOs
- Figure out how to let Anonymous users create new tickets
- Currently looks like a separate form to embed on an appropriate webpage that will then go create a ticket based on provided input
- TST Guidance: Lowering the barrier to creating a Jira account >> creating Anonymous Tickets does not need to be supported
- Figure out how to let Anonymous users create new tickets
- Production site available at https://jira.aetherproject.org
- Confluence
- Test migration appeared to complete successfully
- QUESTION: Can I perform the formal migration from existing Confluence DataServer to Confluence Cloud between 12:00 noon Wednesday(8/21) and 12:00 noon Thursday (8/22) (PDT)?
- Will require read-only Confluence installation at start of process until completion.
- Upon completion, existing wiki.aetherproject.org will point to new site
- Previous site will be maintained for approximately 30 days as an archive/reference site and then hopefully deleted.
- I would hope that people can spend a 30-60 minutes reviewing the site after completion and ensuring things look okay. Hoping to have acceptance of new site by next week's TST.
- Post note on migration to #aether-dev channel
- Jenkins
- No progress as yet
- No roadblocks
- SD-RAN DNS
- No progress from last week
- Jira
- OnRamp
- PR related to k8s configuration pending
- Still debugging monitoring system (metricfunc failing)
- Question to Group: Anyone connected to "Aether Using Nephio" project?
- SD-Core
- Incorporating the static checks for linting with other PRs
- No progress on logging
- Discussion about e2e test on PRs
- Waiting on repo omec-cni, omec-pod-init (to rename)
...
- Aether Jira Proposal: Aether Jira Proposal
- Tweaked proposal and accepted as "good enough to get started"
- Should be ready to play with next week
- Github org renaming: https://docs.github.com/en/organizations/managing-organization-settings/renaming-an-organization
- More representative → Mostly focused on "opennetworkinglab" (should be "aether")
- May break external links, but now is a good time to change since we're creating new docs/pages
- No objection...
- Slack vs Element (https://element.io/)
- Limiting history to 1 year (affects "grandfathered" workspaces like onf-community)
- Proposal: prune onf-community back to "aether" and accept history limit (take up with voltha and p4)
- Code of Conduct ((https://lfprojects.org/policies/code-of-conduct/)
- Look through the LF standard and decide at a future meeting
- Jenkins Status
- ONOS GitHub Actions
- Actions that run for PR (ROC done)
- Actions that run for PR (SD-RAN done)
- Not yet publishing new Docker images (in progress)
- OnRamp Integration tests
- JJB Related: Jobs with same name as the one created earlier, are not getting created again.
- Native Tests: All the Jenkins jobs are not able to go beyond aether-5gc-install step. → "new group" command hanging
- ONOS GitHub Actions
- omec-pod-init and omec-cni repos Gabriel Arrobo
- These repos are in Gerrit → Why? Not updated in years. Move to omec GitHub? → Yes.
- TST Meetings in July → Will remain on the schedule and expect them to happen (with some people missing)
6-18-24
- DCO (instead of CLA) in place Larry Peterson David Ferguson (Deactivated)
- See Org README here: https://github.com/opennetworkinglab
- Will current ONF/CLA be removed from the repos/checks? Gabriel Arrobo
- Add more helpful info about DCO so new users know what do to.
- Org README should explain and link DCO.
- David Ferguson (Deactivated) to look into "augmenting" the error message
- Configuring new Jira Larry Peterson David Ferguson (Deactivated)
- Archiving old Jira
- Proposal
- Projects: SD-Core, SD-RAN, ROC, OnRamp, Smart5G
- Workflow: New, Assigned, Resolved, Reopened (start simple)
- What Fields: Description, Status
- Hopefully will demo at next week's TST
- GitHub default branch should be main instead of master. Ajay Lotan Thakur
- Master is old way; main is the new way (and the default)
- Prefer to have all the same; most currently master
- Decision: leave it as is (expect to be mixed)
- BESS-UPF Helm Charts: Current charts assume symmetry for the access and core interfaces (i.e., same CNI Plug-in for both) Gabriel Arrobo
- Any objection about making the Helm Charts as generic as possible such that each individual CNI can be independently chosen?
- Example: access=macvlan, core=host-device
- But it remains the default that they are the same (currently macvlan); so remains backward compatible
- Will eventually want to expose in OnRamp
- Road map cleanup Ajay Lotan Thakur Gabriel Arrobo
- Priority to take up only SBI upgrade this quarter/end of Sept 2024
- Priority on bug fixes and issues reported by other community members
- See Roadmap for more details
- Work harder to get info to the community / work to attract contributors
- SD-RAN update Woojoong Kim (Deactivated)
- Working with a team of 5 students
- Github action → Almost done (run tests and publish images)
- Scalable RAN Simulator → on going; hopefully done in Jun.
- Dependency update → dependabot has been busy; resolving 1-by-1 (waiting on GitHub Actions)
- OnRamp Update Bilal Saleem
- Separate Management Node → Dealing with local charts was the complication
- Dueling Docker installs → Ansible playbook now checks for existing Docker, per Docker best practices
- Check that multiple gNBs work for data plane
- Working on proposal for incorporating scalable Core into OnRamp
- QA Update Kunal Kapoor
- JJB code commit is done
- Working on updating Readme and streamlining the code as per the current requirement.
- David Ferguson (Deactivated) : Customizing the job as opposed to customizing the build machine (so can dynamically grab and release VMs)
...
- Archive "tiny" (unused) repos [Gabriel]
- As part of release 1.4, several repos were consolidated into the "util" repo
- Do a patch release for Docker images (1.4.1) [Gabriel]
- New Helm Chart also just published
- Expect a PR to update OnRamp
- Working on a blueprint for a DPDK configuration
- SD-Core Roadmap Discussion
- What's the process
- We have an old Google Doc
- Move it in the Wiki
- Community contributes
- How do we move past Release 15?
- Start with OpenAPI spec for R17
- Generate code (and test/fix)
- Results in the wish-list
- What's the process
- SD-RAN Status
- Training new contributors
- To document in a new wiki page for future onboarding
- Jenkins to Github action conversion for SD-RAN is mostly done
- Expect to complete this week
- Training new contributors
- ROC Status
- PR now merged
- New Helm Chart published by GitHub Action
- New version (2.1.36) was published => it did not exist (but has been used by OnRamp)
- What gives???
- OnRamp Status
- Bilal looking into two issues
- Docker issue when deploying in CloudLab
- Local helm charts on master node
- JJB Status
- Testing on AWS (Goal is to be re-producible)
- Automate Jenkins job creation using JJB and definitions from git repo
- Seed Job on builtin node to pull the latest changes in aether-jenkins repo - Done.
- Cron job to update the jobs in Jenkins.
- ...
...
- LF Transition Update
- Mailing lists now up: tst@, dev@, announce@lists.aetherproject.org
- Instructions for joining lists: Visit https://lists.aetherproject.org/g/TST (dev | announce) to join
- QA Status
- Integrating UERANSIM; expect a PR tomorrow.
- Canonical’s Juju-managed Aether 5G Core (Mark Beierland Serdar Vural)
- Slice Deck: LF Aether TSC (19 March 2024) - Charmed 5G Core.pdf
- Source Code:
- https://github.com/canonical/sdcore-amf-k8s-operator
- https://github.com/canonical/sdcore-ausf-k8s-operator
- https://github.com/canonical/sdcore-nrf-k8s-operator
- https://github.com/canonical/sdcore-nssf-k8s-operator
- https://github.com/canonical/sdcore-pcf-k8s-operator
- https://github.com/canonical/sdcore-smf-k8s-operator
- https://github.com/canonical/sdcore-udm-k8s-operator
- https://github.com/canonical/sdcore-udr-k8s-operator
- https://github.com/canonical/sdcore-upf-k8s-operator
- https://github.com/canonical/sdcore-webui-k8s-operator
- Documentation: https://canonical-charmed-5g.readthedocs-hosted.com/en/1.3-beta_a/
- For lab work using Aether → Please add "abstract/summary" + related links to Research page on wiki.
...