Agenda & Meeting Notes

Owned by Larry Peterson
Last updated: Dec 18, 2024 by Gabriel Arrobo


For an archive of meeting notes pre-dating the Aether Project moving from ONF to LF, click here.

If you need technical help from the Linux Foundation IT staff, please submit a ticket (typically requesting a Project Service). When creating a ticket:

  • Component = 'Release Engineering'
  • Product = Aether

1-7-25 (Topics for Next Meeting)

  • ...
  • SD-Core
    • Need to review/update maintainer's group in the omec-project organization (Gabriel)
      • Add Ghislain to maintainer's group and should we still require a single approver, or should we increase to 2 approvers?
    • Do we need to keep creating "per-PR" images? There are not used anywhere (Gabriel)


12-17-24

  • Remaining meetings: December 24 and December 31 - keep or cancel?
    • To cancel. Next meeting Jan 7th. 
  • ROC
    • GitHub Actions to publish ROC artifacts. Status?
      • Have merged latest changes .github → to publish helm charts
      • Next step is to test run in an example repo 
      • Avoided being "overly generic" → targeting ROC in onosproject repos, but can be adapted
      • Publishing images will be next
      • Want to port to the omec-project repos once done (related to other Jira ticket)
    • Why it is not possible to configure OP (or OPc) and Key through GUI? (Gabriel)
  • Infrastructure
    • Jenkins performance/stability. Any update?
    • Blocked on AWS credentials
  • OnRamp
    • Version control: It would be good to tag versions/releases such that it is easier to provide support to end users (Gabriel)
      • Should be done.
      • Challenges
        • Need well-defined release process => Need dependable testing of blueprints.
          • Releases paced by "weakest" blueprint.
          • E.g., Latest SD-Core release gated by having all the blueprints work correctly.
        • Today, OnRamp releases ~ SD-Core Releases
    • Subnet variables working and ready for PR (see subnets branch of various repos)
    • srsRAN Jenkins job being debugged
    • Process to add other blueprints (e.g., Confidential compute, N3IWF, etc.)? (Gabriel)
  • SD-RAN (Woojoong)
    • Release blockers
      • SD-Core chart 1.4 has an issue in RiaB (4G)
        • It pointed aether helm chart, which seems to be removed → updated it with sdcore-helm-charts
        • It was used very old SD-Core chart → bumped up the version to 1.4.0
        • UPF was facing OOMKilled → fixed by doubling memory limit
        • Too many open files issue in MME and PCRF → set fs.file-max
        • PR: https://github.com/onosproject/sdran-in-a-box/pull/207
      • 4G hardware document (4G)
        • Out-of-dated → updating now
    • What is the plan to merge currently open PRs from Dependabot? (Gabriel)
      • High priority for next release, which hope to start working on after the holidays
    • Why do we maintain/keep RiaB instead of OnRamp? (Gabriel)
      • RiaB currently includes 4G elements that are not supported by OnRamp
  • SD-Core
    • Update GitHub Actions to properly handle tag creation when PRs are towards release branches (e.g., rel-1.5)
    • Continue effort to move code to 3GPP Release 18
      • Created models and APIs for some specs including Nnrf_NFDiscovery that is used in the nrf_cache
  • ARA
    • SD-Core
      • Currently checking logs to understand the UE context release issue (details posted on on-ramp slack channel)
      • Almost done debugging
    • SD-RAN
      • No updates (Tianyi on travel)
    • Blueprints
      • In progress → One for Ericsson radio and one for OAI radio
      • Could include ARA-specifics (e.g., for UEs, including experiments)
      • Intent is end-to-end deployment blueprints
        • related to OnRamp releases since more blueprints => higher hurdle
        • Need "owner" for every blueprint; some may be "core" and others "auxiliary"

12-10-24

  • TST Election Results
    • Welcome Bilal, Ghislain, and Tianyi (welcome back Gabriel)
    • Much thanks to Andy, Scott, and Shahbaz
  • Infrastructure
    • Provenance of registry.aetherproject.org
      • Gabriel has admin for sdcore
        • Incremental images pushed to Aether Registry
        • Officially released images pushed to Docker Hub
      • Overall admin unknown
      • Have now identified AWS account
      • There are other projects that are no longer related 
      • See Jenkins issue below
      • In the past this registry wasn't 100% reliable
    • GitHub Action consolidation (reusable workflows) for omec-project
      • work is underway
    • GitHub Actions for publishing ROC artifacts
      • To get input from Gabriel and Sean
      • Repos: roc-helm-charts, onos-config, onos-topo, aether-models, onos-cli, onos-api, config-models, onos-helm-charts, build-tools, aether-roc-[api,gui], onos-lib-go, onos-test, sdcore-adapter
    • Jenkins performance is poor → image download seems to be an issue
  • OnRamp
    • Cleaning up subnets
      • Variables for core_subnet (192.168.250.0/24) & access_subnet (192.168.252.0/24}
      • Works for gNBsim
      • Testing for OAI, UERANSIM and srsRAN in progress
    • Update namespace for SD-Core to "aether-5gc"
  • SD-Core
    • Reviewed and merged ~40 PRs opened by Dependabot
    • Improvements to code by properly handling possible Panics (AMF, NRF)
    • Continue working on 3GPP release upgrade (Release 18)
  • SD-RAN (update posted to Slack)
    • SD-RAN Engineering team showcased SD-RAN and Aether-OnRamp in Korean conference - went well!
    • SD-RAN release
      • We bumped up the sd-core chart but it's not working with 4G OAI with SD-RAN in a Box.
      • We are investigating it - this is the release blocker, which I am mostly trying to fix..
  • ARA Deployments
    • SD-RAN
      • Studying the RIC/SM/xApps to see how them can contribute to rural application
    • SD-Core
      • Started rolling out SIM cards for field-deployed UEs to work with SD-Core
      • UE context releases after sometime and unable to register afterwards.
        • Happens after a few hours, even with active traffic; can't reattached
        • With Ericsson gNB (not observed in the past when used their core)
        • Collecting more logs to help debug

12-03-24

  • TST Election
    • Eligible voter list (mostly) complete
    • Nominations due by Friday
  • Infrastructure
  • SD-Core
    • Merged PRs (UPF and SMF) related to TEID allocation
    • Currently working on enabling cert-manager as a sub-chart for the SD-Core Helm Charts
    • Continue working on 3GPP release upgrade (Release 18)
    • Scalability PRs still in progress
  • SD-RAN
    • Penn State reporting security risks  in SD-RAN (also SD-Core); see report
  • OnRamp
    • OnRamp is now using the latest version of SD-Core Helm Charts (v2.0.0)
    • srsRAN Blueprint (simulation mode only) now available (integration tests & documentation still pending)
    • Any thoughts on renaming the default branch to use current GitHub name (main) instead of master for OnRamp and its dependencies/sub-modules?
    • Are the "dependencies" such as versions of K8s, Helm, etc. frequently updated?
    • Why not accordingly update the name for SD-Core namespace instead of using legacy name (omec)?
    • Integration tests (Jenkins) still plagued by timing issues (e.g., only 4 of 5 UE succeed).
  • ARA
    • SD-Core w/ Ericsson RAN
      • Tested with 4 E/// BSes
      • Fixed IP assignment to specific UEs possible?

11-26-24

  • No meeting

11-19-24

  • TST Election
    • Soliciting nominations
    • Determining who qualifies to vote
  • Infrastructure
    • Issue with srsRAN Docker repo
  • SD-Core
    • Process for accepting PRs when optional components get "affected" (stop working)
    • Chunk-tizing PRs from Purdue for scaling
    • Propose and vet designs before submitting PRs (
      • especially when need to take e2e issues into consideration
  • SD-RAN
    • Aether On-Ramp has 1.5 SD-RAN helm chart now
    • Fixed user doc site fully! - multiversion had an issue; it only showed master but now it showed old version too.
    • Documentation work on going
      • Release note draft ready - will ask a review sometime this week
      • HW installation - engineering team - Hyunmin - is leading to write it up
      • Aether On-Ramp for SD-RAN - draft is ready
    • Engineering team is debugging E2 decoding issue when testing with Viavi E2 simulator
    • Starting to turn focus to next release (see wiki)
  • OnRamp
    • Process for testing SD Core Helm Charts v2.0.0
    • Completed srsRAN blueprint (simulation mode only)
      • We (Aether) are creating the required Docker images 
      • Address some intermittent issues (e.g., AMF crashes)
      • Add support for gNB w/ USRP
    • Cleaning up subnet configuration & usage (in progress)
      • Document the Linux bridges being used
    • Ansible: host.ini (sudo password)
  • ARA Deployments
    • SD-RAN:
      • Working in outdoor.
      • Checking how the RIC supports URLLC, slicing and spectrum sharing
    • SD-Core:
      • Throughput stable with E/// RAN
        • Tweaked Core-to-gNB transport (best guess at the moment)
      • SD-Core pushing around 850Mbps to E/// RAN and UE getting 600Mbps on avg.
        • May have been environmental factors; will retest
      • Enable DPDK and test throughput again.
      • Will also test with multiple gNBs

11-12-24

  • SD-Core
    • Updated Helm Charts with several improvements (use yaml extension, configurable path for config files, among other)
    • SMF/UPF: TEID allocation
    • IOS-MCN
      • Proposal for multiple DNNs
      • SD-Core reset: AMP
  • Aether OnRamp
    • srsRAN blueprint in process of merging
  • ARA
    • SD-RAN 
      • Working on outdoor testing of SD-RAN with OAI + USRP SDR.
    • SD-Core w/ Ericsson RAN
      • ARA team, IL and Ericsson will have a debugging session tomorrow

11-5-24

  • TST Elections
  • Infrastructure
    • Sphinx documentation (hosted on static.opennetworking.org basically)
      • Migrate to ReadTheDocs?  (SaaS for Sphinx documentation)
      • link: https://about.readthedocs.com/
      • Standard throughout LF
      • Community (free with ads), Gold (no ads, $5.00/month)
      • Alternative is to publish on GitHub (llp)
    • DNS
      • In process of moving and migrating off of referencing *.opennetworking.org sites
        • Trying to avoid downtime or docs sites being unavailable during transition.
    • LF Insight needs to be updated to report stats for all three GitHub Orgs
  • Aether OnRamp
    • Jenkins jobs now successfully running for all Blueprints
    • Rich debugging opportunity – looking for volunteers
  • IOS/MCN Presentation
    • Overview Deck
  • SD-Core
    • Enabled secure communication (TLS/https) for SBI by default in Helm Charts
      • Pending: i) Secure GRPC communication; ii) Use a CA instead of self-sign certificates
    • Alignment for configuration across all NFs (Docker image, configuration arguments/flag, parametrize path for config file, etc.)
  • ARA
    • SD-RAN
      • Working in sandbox with OAI and USRP SDR; moving to outdoor testing.

    • SD-Core w/ Ericsson RAN

      • Working with Gabriel and Suresh to get a common time to have a meeting with Ericsson team to troubleshoot throughput instability issue observed.
  • ...

10-29-24

Does anybody use:

wiki.opencord.org
static.opencord.org
charts.opencord.org (needs attention --llp)
guide.opencord.org
opencord.org
static.opencord.org
wiki-archive.opencord.org
www.opencord.org
    • Slack permissions → Needs cleanup, but only workspace owner can demote admins
  • OnRamp
    • Blueprint Quick Reference
    • All Jenkins jobs are working, except Multi-UPF
    • Loading Models into ROC remains flaky (FYI)
    • When ROC controls the core, after a Core reset (restarting all the pods), e2e tests fail (performance metrics also reset)
  • SD-RAN
    • SD-RAN Doc repo now has github action to publish the doc!
    • 1.5 Doc is WIP especially for hardware installation and release not (hope it will be done this week)
    • Will announce internally, then do a public announcement (blog, webinar, etc)
    • Working on onos-e2t integration with Viavi simulator actively. We got the error message and now try to fix it.
    • Start planning 1.6 release
      • Bump up dependencies' versions and go version
      • Re-design APIs
      • CCC service model (PR ready, but not fully tested yet)
      • Viavi simulator integration
      • RAN sim v2.0
      • OAI 5G integration (use latest OAI code)
      • non-RT RIC integration (1.6 - just for PoC)
      • User documentation
  • SD-Core
    • Enable ZUC algorithm (NEA3, NIA3) at NAS level
    • Enable HTTPS/TLS for SBI interface in sdcore-helm-charts
      • Parametrize path for certificates in NFs
    • NFs go into an infinity loop when the webui pod is restarted
      • Canonical provided a solution, which was merged in most of the NR (PR for UDR still pending)
    • Clean up deprecated linters from golangci
    • Enable authentication and authorization for WEBUI
    • Breaking Bilal's big PR into chunks and working through them
  • ARA
    • SD-RAN
      • Work on SD-RAN with OAI (2022.w51). Connected the RIC to OAI gNB and SDR but solving issue to connect to OAI nrUE.
    • Ericsson GNB Performance Investigation: → We tested various use cases with different core and find below our observation (all tests are UDP tests).
      • iperf with UERANSIM against aether SD-Core:
        UL throughput: 986 Mbps
        DL throughput: 986 Mbps
      • iperf with Ericsson GNB against aether SD-Core:
        UL throughput: 111 Mbps
        DL throughput: 144 Mbps
      • iperf with Ericsson GNB against open5GS:
        UL throughput: 109 Mbps
        DL throughput: 143 Mbps
      • We tested using fast.com (which uses TCP) and we were getting around 144Mbps.
      • Initially get expected xput with SD-Core, but a packet drop impacts the xput rate (that doesn't happen w/ Ericsson core)
  • TST Elections → Coming soon.

10-22-24  (Today's Agenda)

  • Infrastructure Update
    • Expired certificates updated → How do we keep this from happening again?
      • Warning to be sent to aether-tst channel 
      • By using local Helm Charts, OnRamp can avoid this issue
    • DockerHub rate limit → Now impacting daily integration tests
      • Publish all images to Aether Registry (official releases still go to DockerHub)
      • But... have to keep the Aether Registry available
      • To put together a specific plan of action (daf, llp, gabriel)
    • DNS
  • OnRamp Update
    • Refactoring PR to be merged later today → Watch for updates to documentation
    • UERANSIM blueprint includes data plane test
    • Why do we need Helm Charts from opencord when deploying the 5GC?
      • I don't think we do; plan to remove.
    • Re-org docs to highlight blueprints
  • SD-RAN Update
    • Adding Github action to SD-RAN doc repo
    • Updating SD-RAN doc - especially hardware installation part as it is out-of date
    • Latest Helm Chart (1.4.x) is actually for v1.5
      • going to re-number as 1.5.0 Helm Chart version to match release
    • O-RAN plugfest - Viavi integration; kick-off meeting this week
  • SD-Core Update
    • Renamed default branch for all repos to "main"
      • Some repos already used "main" as default
      • For the other repos, everything got automatically updated (e.g., PRs)
    • UPF-Adapter code update in progress (enable metrics)
      • Helm Charts are already updated
    • Review all outstanding PRs
    • gNBsim paper under review
  • ROC Update
    • GitHub Actions?
    • Still pending
  • ARA Update
    • SD-RAN deployment 
      • Begin setting up SD-RAN, progressing toward running OAI 2022.w51 with SD-RAN.
    • SD-CORE deployment
      • Trying to improve throughput from Ericsson gNB
        • Expecting 700-820Mbps
      • Suresh at Intel Labs is focused on performance
        • He's seeing no issues (getting around 1Gbps)
        • Connecting Suresh with Joshua on Slack
        • Easiest to troubleshoot using UERANSIM
    • Exploring SMART-5G & Rakuten POC effort (09-24-24 meeting notes) 
    • Miscellaneous items
      • Interest in "ARA Blueprint" advised by Larry 
      • Plugfest at ARA OTIC (O-RAN Testing Centers)
      • June 2025 gathering of wireless and AgTech industry leaders (via NTIA ACCoRD project & likely AraFest'25)


10-15-24

  • Infrastructure
    • Docker Hub organizations (residual from ONF days)
    • Limit of three seats per org
      • Need one for "lfreleng"
      • Need "onfauto → currently pushes SD-Core (and other) images
      • Open Third Seat  → assign to a TST member (currently Gabriel)
    • Three "quiet" orgs → xosproject & onfinfra (remove), opencord (check with Voltha)
    • Five orgs created but never used (keep "aetherproject" & "openaether" remove the other three)
  • SD-Core Update
    • Release new Helm Charts using the latest versions of the images, including the upfadapter image
      • Code for the upfadapter was removed from the smf repo
    • Reviewed and merged most of the open PRs (Currently, 14 PRs open: 5 are drafts, 5 dependabot PRs have issues due to protobuf update/version)
    • Daf will help us renaming the default branch to "main" and update the GHAs to point to "main"
    • omec-pod-init and omec-cni repos still pending to create
  • OnRamp Update
    • Pipelines for integration tests now in GitHub
    • Stability remains an issue
      • DockerHub rate limits
        • Anonymous users: 100 pulls per 6 hours per IP address
        • Authenticated users: 200 pulls per 6 hours
        • Docker Pro and Team subscribers: Up to 5,000 pulls per 24 hours 
      • VM resources?
        • Base (4vcpu+16GB). Trying (8vcpu+32GB)
      • TIme to download images from DockerHub?
        • Could shift to Aether Registry (on a VM inside our AWS account)
        • Docs, Registry, Jenkins, Docker Builder/Executor (may not need since we've moved off Jenkins CI tools)
    • ueransim blueprint is now using the latest version of SD-Core Helm Charts
      • Issues starting the gNB and UE as Linux processes? → Was working at one time, but...
      • The Ansible playbook is flakey(did work for 1 of 3 instances – UE/gNB pair of processes)
      • ARA also seeing problems
    • srsRAN blueprint status?
  • SD-RAN Update
    • ...
  • ARA Update
    • Waiting on v1.5 release of SD-RAN
    • Still seeing throughput issue for Ericsson gNB and SD-Core
  • ...

10-8-24

  • SD-Core Update
    • Immediate Goal →  Merge Scalability PR
    • Replacing "logrus" with "zap" logger was completed
    • Created new images
    • Settle on main rather than master → so consistent for automation
    • There are currently ~45 PRs open (30+ dependabot), rest from Canonical
  • SD-RAN Update
    • Immediate Goal → Complete v1.5 release
      • Documentation CI was not configured - figuring out how to do it (checking Aether doc repo)
        • Both SD-Core and Aether have moved to GitHub actions
        • Main change to docs is the release notes
      • No other engineering task other than documentation
    • Try to participate O-RAN plugfest fall 2024 (hosted by LG U+ - Korean TelCo company)
      • Topic: Viavi RIC tester & SD-RAN E2T integration
    • Participated ICT Challenge 2024 (Korean conference)
      • Showcased SD-RAN (potential 1.5 release code) E2E test with 4G SD-Core, OAI 4G, USRP working
    • Submitted one conference paper (Korean domestic conference KICS 2024 Fall) - SDRAN E2E test topic
    • Wish List: Combined SD-RAN + OAI 5G blueprint (part of v1.6)
      • Open Question: Latest OAI code vs Older version with our E2 Agent
  • OnRamp Update
    • Immediate Goal → Reliable daily builds for all blueprints
    • Auditing (and cleaning up) integration tests
    • Improvements to UERANSIM blueprint
      • Still need to debug issue with latest charts
    • Other "robustness" improvements in progress
    • Wish List: an srsRAN blueprint (modeled after the OAI blueprint)
  • ROC Update
    • Immediate Goal → Reproduce CI tooling
    • Plan is to jump straight to GitHub Actions (Publish is still missing)
  • ARA Update
    • Immediate Goal → Deploy Aether in various scenarios
    • Waiting on SD-RAN v1.5, then will integrate with OAI 5G
    • Working on Ericsson gNB ↔ SD-Core 
    • Participating in ACCoRD (NTIA funded) as "neutral" party (Radisys gNB + SD-Core)
  • ...

10-1-24

  • SD-RAN Status
    • SD-RAN 1.5 code is all ready
      • OnRamp has been upgraded to deploy
    • SD-RAN doc could not be published. Checking the CI.
    • Now Planning 1.6 release
  • Infra Status
    • Jenkins
      • Original "Static VM" jobs have all been disabled
      • Full set of "Dynamic VM" jobs are being debugged
    • ...
  • SD-Core Status
    • Scalability PRs are being evaluated
      • GitHub Actions flagging some issues
    • Start replacing Logrus with zap logger
    • Fixed issue with gnbsim profile4 (parallel execution)
    • Several other patches accepted this last week
      • From Canonical
      • From IOS
    • Release Plans
      • Next: Scale + Logger
      • Future: Rel 18 upgrades
  • OnRamp Status
    • SD-RAN Blueprint updated for v1.5
    • UERANSIM Blueprint has been refactored
      • Need to update the documentation
    • Additional refactoring still in-progress
  • Radio Interoperability
    • Ericsson (AIR 6488) working, but suboptimal throughput
  • ...

9-24-24

  • SD-Core
    • Pending "Scalable Core" PR (Bilal)
      • NRF caching (remove bottleneck)
      • Enhanced load balancing (UE-based caching)
      • Also includes several latency/bug fixes
      • Touches nearly all Network Functions
      • Plan is to review with Gabriel and Ajay
      • Changes on GitLab; will issue PR to GitHub
        • Gabriel will postpone log-related changes until after merge
    • Progress connecting Ericsson gNB to SD-Core (Gabriel, Joshua)
      • Fixed issue with NGAP field
      • Now have N2 connection → Have successfully established PDU session
      • Now tripping over User Plane issues 
    • Working on Release upgrade (Gabriel)
    • There has been a steady stream of PRs from IOS team
  • LF News
    • free5gc project has joined LF
    • Talk to Timon about "harmonization" opportunities
    • Coincidently, today we finished "repo independence" from original fork
    • API for Release upgrades could be a useful point of cooperation, but free5gc uses their private code generation tools
  • OnRamp
    • Issues with ueransim blueprint (Gabriel)
      • There might be "secret sauce" in Jenkins job
    • Kubernetes config PR committed (see Allocating CPU Cores)
    • Modest refactoring of Ansible variables still in progress
    • OAI Blueprint → Improperly set config flag (Joshua)
  • Infra Update
    • Reminder to use Jira to report bugs and request features
    • 2-server Jenkins jobs still not running correctly → Might be running (changing output to determine)
    • Working on stability of other jobs
  • SD-RAN Update
    • New Helm Charts released: 1.4.143
    • Have not heard any news on 1.5 release notes
  • O-RAN SMO / non-RT RIC integration? (in addition to Aether Management Plane?)
    • Status & roadmap for SD-RAN A1 interface? O1 interface?
      • Expected people effort needed (if any)? 
    • Candidate SMO / non-RT RIC that is more compatible with SD-RAN? OSC SMO? 
    • Aether Strategy has been to use ROC (AMP) as "RAN SMO"
      • Prototyping work was done, but not carried forward
        • PRs exist, but not merged
        • Depend on CCC Service model, which is yet-to-be merged.
      • The A1 interface for SD-RAN does exist
    • There is also SMART-5G work that as been leveraging open source SMO, rApps and O1/A1 interfaces: 
  • Miscl. 
    • free5gc & commercial version

9-17-24  (No Meeting)

9-10-24

  • SD-Core Development Priorities
    • Reminder: Current Roadmap 
    • Upgrade to Release 18
    • ARA Focus
      • Get SD-Core working with OAI (done, being demo'ed at OAI Summit)
      • Also want to demo Core with other radios
        • e.g., Working with Ericsson radios with Aether Core
      • Scalable Control Plane
      • Research more focused on SD-RAN topics
        • Scheduling & Resource allocation issues
        • Low latency & power control)
        • Working on xApps
    • Canonical Priorities
      • Keep up with 3GPP
      • Slicing support
      • Working with other radios (e.g., OAI)
      • Canonical is contributing
        • additional integration mechanisms
        • reliability/hardening (e.g., restarting)
        • removing backdoor dependencies of NF on database
    • ...
  • SD-RAN v1.5 Release Status
    • Latest SDRAN charts test done - all working!
    • Publishing the latest Helm chart now.
    • Finalizing the documentation
  • Infra Update
    • Jira access
      • ALL product work, new features, or issues should be reported and tracked on the Project Aether site: 
      • Other  generic Infrastructure issues:
      • Note – While I am the primary infrastructure engineer for Aether, other members of the LF team may fill in or back up at times.   This is why generic infrastructure issues should be filed in the LF Jira, rather than the Aether-specific one.
    • Jenkins reduction
      • Jobs significantly scaled down ...  
      • All Aether OnRamp jobs EXCEPT for 2Servers-22.04-default-charts dynamic
      • Expect to have 2Servers-22.04-default-charts this evening dynamic
      • What are necessary ROC CI jobs
    • Jira/Confluence
      • Problems with initial access have been seen, but seem to go away after 2 or three attempts
      • Any issues with access?
    • Open Tickets::
      • IT-27166 - Jira Dashboard
      • IT-27178 - Jenkins Clena Up: OnRamp Tests
      • IT-27179 - Jenkins Clean Up: ROC API Tests
      • IT-27105 - Bring omec-cni and omec-pod-init from Gerrit to Github (OMEC)
      • IT-27168 - Forwarded emails being discarded by gmail due to DMARC
      • IT-27116 - Discourse Forum for P4 timing out for new logins
  • OnRamp Update
    • Updated k82-related PR expected soon
    • Global "aether" section of vars file in-progress
    • New "Blueprint Guidelines" now in Doc.

9-3-24

  • New User Communities
  • Infrastructure
  • SD-Core
    • Created upf-adaptor repo (moved out of smf); fixes metric problem
    • Found issues with gNBsim stats
    • Added dependency check of kafka to amf and smf
    • Merged patches from IOS-MCN; other "proposals" being discussed
    • Issues trying to run gNBsim against Magma (similar issues with OAI Core)
      • related to service request handling
    • No progress on Rel-18 upgrade
  • SD-RAN
    • Not much progress on v1.5 release
    • Shared OAI code with ISU; should work now
    • Testing in prep for release the end of this week
    • Writing documentation for release (how to use OnRamp for SD-RAN)
    • Once 1.5 is ready, will start integrating CCC E2 Service Model (merged into master)
  • OnRamp
    • PR related to k8s configuration still pending
    • Evaluating "global" section & design guidelines

8-27-24 

  • SD-RAN
    • Woojoong will provide an update later
      • Will update on Sept W1 deadline
    • Iowa state univ
      • could not build the E2 Agent from OAI from 2022w51 tag
      • Kushal pushed a patch - could build E2 RIC agent and 5G OAI with it
      • Should help for 1.5


  • SD-Core
    • Created new images (with Canonical)
    • Published new helm charts
    • Gabriel has tested with OnRamp
      • Would like Larry to verify them on OnRamp
      • Test them on Jenkins jobs
    • One of the fixes in GNB Sim allowed Bilal to move forward
      • Found the next issue - still investigating
      • Had not tested with Magma
      • Also testing GNB Sim across multiple 5G cores
        • e.g Free 5GC, Open5G
    • Fixed a metrics bug in SD-Core by downgrading Kafka
      • Will look to upgrade Kafka and Mongo to latest next
    • Replacing Logrus with Zap logger
      • Canonical may be able to help with this
    • Discussion about e2e test on PRs
      • No progress on this
    • Waiting on repo omec-cni, omec-pod-init (to rename)
      • Gabriel waiting on Daf to create these GitHub repos


  • OnRamp
    • PR related to k8s configuration pending - meeting with Gabriel, Suresh and Larry to sync up on this



  • ROC
    • No progress on updating Wiki - Sean to continue on this


8-20-24

  • SD-RAN
    • Working on v1.5 release
      • Working on documentation and release notes
      • To include OAI 5G RAN 
        • includes "ONF" E2 agent (not OAI's)
        • implies need to rebuild OAI images (since E2 agent is statically included)
      • Not include update to golang 
      • Expect announce 1st week of September
      • E2SM → Will add O-RAN RC Service Model 
        • Backward compatible with the old ONF Service Model
  • Infrastructure
    • Jira
      • Production site available at https://jira.aetherproject.org
        • Still working out Anonymous creation of issues
      • Permission Question:
        • Currently four levels of access
          1. Anonymous (users without Atlassian accounts or not logged into them) can Browse and Read existing Issues but not modify anything
          2. Any Logged in user can basically change almost any field in an issue, and invite people to join the site and become a user
            1. When invited, specific domains are automatically approved when invitee makes the request
            2. Basically, any  TSC user's domain (that is NOT public (ie, not gmail, yahoo, aol, ...))] is currently automatically approved.
          3. aether-jira-admin users can update group and user permissions
          4. admin can do anything.
        • QUESTION: Any need now to restrict (beyond being logged in as an approved Aether Jira user) things like:
          • Transitioning a ticket between states (including closing)
          • Mucking with assignee
          • editting fields beyond comments
          • → Taking a wait-and-see approach
      • Known TODOs
        • Figure out how to let Anonymous users create new tickets
          • Currently looks like a separate form to embed on an appropriate webpage that will then go create a ticket based on provided input
        • TST Guidance: Lowering the barrier to creating a Jira account >> creating Anonymous Tickets does not need to be supported
    • Confluence
      • Test migration appeared to complete successfully
      • QUESTION: Can I perform the formal migration from existing Confluence DataServer to Confluence Cloud between 12:00 noon Wednesday(8/21) and 12:00 noon Thursday (8/22) (PDT)?
        • Will require read-only Confluence installation at start of process until completion.
        • Upon completion, existing wiki.aetherproject.org will point to new site
          • Previous site will be maintained for approximately 30 days as an archive/reference site and then hopefully deleted.
        • I would hope that people can spend a 30-60 minutes reviewing the site after completion and ensuring things look okay.   Hoping to have acceptance of new site by next week's TST.
        • Post note on migration to #aether-dev channel
    • Jenkins
      • No progress as yet
      • No roadblocks
    • SD-RAN DNS
      • No progress from last week
  • OnRamp
    • PR related to k8s configuration pending
    • Still debugging monitoring system (metricfunc failing)
  • Question to Group: Anyone connected to "Aether Using Nephio" project?
  • SD-Core
    • Incorporating the static checks for linting with other PRs
    • No progress on logging
    • Discussion about e2e test on PRs
    • Waiting on repo omec-cni, omec-pod-init (to rename)

8-13-24

  • SD-Core
    • Check with Ajay again about an SD Core deployment mode (below)
    • Status update:
      • Removed in gnbsim dependency from the CommonConsumerTestData. There were a few functions calling the CommonConsumerTestData  but these functions were not used. So, CommonConsumerDataTest  repo is now archived
    • golangci-lint not detecting all issues. For example, staticcheck  is enable in golangci  and it successfully completes. However, if we run staticcheck  by itself, it finds several issues that are not found through golangci 
    • We use logrus  (https://github.com/sirupsen/logrus) for logging, where the last commit is from June 2023. This might mean that the project is inactive (none is actively maintaining it). Based on this, I explore a few options and zap (https://github.com/uber-go/zap) seems to be a good alternative. As per its description zap is "Blazing fast, structured, leveled logging in Go". My suggestion would be to replace the logger. What are your thoughts? If agree, who can volunteer to work on this?
    • It would be good to check whether golanci-lint catches issues when using zap
    • GitHub Action (or Jenkins job) to run E2E tests for PR verification/validation
      • This could mean the use of a simplified version of AiaB
    • Bring omec-cni  and omec-pod-init from Gerrit to GitHub (omec-project organization)
    • A "Scalability" PR is in progress
    • Metrics currently not being displayed by Grafana → dashboard spec has fallen out-of-date
  • OnRamp
  • Infrastructure
    • Jira
      • Setting up production site – anticipate availability EOD Wed.
        • Site will be lf-aether.atlassian.net → jira.aetherproject.org
      • Accounts will be initially all invited members of the TSC
        • Please file a ticket (in lf-aether-test.atlassian.net) if you want to pre-seed other accounts
    • Confluence
      • Migration to Cloud (from existing LF-hosted site)
        • Test migration of existing content (from current wiki.aetherproject.org) this week
          • No downtime
          • Will request (on #aether-dev?) BETTER SUGGESTION? (Yes – largest subscription) for folks to review test site for validity
        • Ideally, request production migration next week's TSC
        • Upon approval, site will be offline approximately 4 hours
        • No functional changes anticipated.

8-6-24 

  • SD-Core
    • Should the sdcore-helm-charts work "out-of-the-box"? That is, should the users be able to deploy the SD-Core by running "helm install" command without having to use/rely on Aether-OnRamp (or AiaB)? [GA]
      • There are somethings need to be modified first e.g. UPF is deployed in DPDK mode default (most performant)
      • It depends on your hardware - there are a number of pre-requisites for this.
      • AiaB and OnRamp deploys in AF_Packet mode (in additional values file)
      • Q for Ajay & SD-Core team - should the chart use AF_Packet by default - we should strive for a positive experience when bringing it up regardless of method
    • Moving towards SD-Core 3GPP Release 18 (Gabriel)
      •  Implementing on SD-Core Roadmap
      • Executing on parts 1.a and 1.b among others
  • SD-RAN
    • Release 1.5 - in Aug with no dependency update - it is too much tightly coupled with other ONOS components especially for onos-api. (WK)
      • SD-RAN components are on Go 1.17 to 1.19
      • Option 1
        • Continue trying to update everything. This is becoming increasing difficult because of common dependencies
      • Option 2
        • Can we separate SD-RAN ric from uONOS under Aether or separate Github workspace? - in that case, we could say SD-RAN inherits a design/code from uONOS.
          • While doing that, fix e2sm and add new ransim.
          • Prevents breaking other components
      • Woojong will talk with Larry after SD-RAN 1.5 release to try to make a decision
    • Talked with Awias today (Prof in Univ of Glasglow): great discussion and will keep doing it to find out the new research topic.
      • ARs: share the SD-RAN tutorials to him.
      • Possible research topics
        • Secure comm
        • AI/ML based self healing of RAN
  • LF collaboration updates (Timon)
    • RIC evaluation project kicked off
      • Funded, Rimedo is contractor. Has kicked off. Final report in December 24
    • Discussing making Aether onramp 'blueprints' nest inside the notion of LF 5g-superblueprints to provide core and various RAN options
      • LF wants to consolidate around SD-Core - the core of choice for LF (replacing Magma from Meta which is waning)
      • 5g-superblueprints are very broad in scope
        • Many different topics e.g. Edge, Cloud
      • We should try to package a proposal on embedding SD-Core (and Aether OnRamp (as a blueprint)) in to these 5g-super-blueprints and present this to LFN TST
  • ROC
    • Continue updating Wiki on the update of ROC model (Sean)
  • Infra updates
    • No new reports - Daf not present


7-30-24

  • Infrastructure updates:
    • Jira Demo
      • No issues reported in Jira setup.
      • Did anybody play around?
    • Jenkins 
      • 12/13 QuickStart jobs converted to Dynamic
      • Yesterday - 3 failures — Today, 12  
        • gnbsim summary not being generated → likely due to VM setup not being stably finished
      • NOTE: QS-2004 and QS2204 views — *EC2 jobs are dynamic
    • Docs
      • CNAMES for docs.sdcore.aetherproject.org & docs.sdran.aetherproject.org → Done
      • What's next to serve content from those names?
  • SD-Core Status
    • Coordinate interface changes
    • SMF updated to use N4 interface (moving off OMEC's PFCP)
  • SD-RAN Status
    • Aether nRT-RIC not 100% compatible with O-RAN E2 Service Models
    • Multiple flavors and customizations of E2SM
    • Also impacted by ASN.1 encoder (ambiguous standard)
    • E2 Agent needs to account for this / E2 Agent currently statically linked into OAI images
  • ROC Status
    • Document the e2e process for modifying the Aether API
    • GitHub Action to publish → Big issue is testing before publishing
  • OnRamp Status
    • OAI blueprint done (documentation available later today)
      • pulling in docker images from dev branch
      • some cleanup to do (deps/aether-oai → deps/oai)
      • upper/lower-case mix on make targets
      • two license files
    • SD-RAN + OAI blueprint discussion
      • optionally include ORAN service models (E2 agent)
      • need to substitute our own E2 agent (inside Intel, but not yet released)
      • to be include in v1.5 release
      • may need to build and publish our own OAI images
    • Should be possible to add srsRAN in pretty much the same way
    • Scalability blueprint proposal? → ask for comments this week
      • Timing of scalability updates with Rel-18 updates

7-23-24

  • Infrastructure Updates
    • Jira Demo
      • Testing playground → lf-aether-test.atlassian.net
      • All should have invites, if not, please Slack me
      • Component:  "Jira Setup" for issues/features/requests
        • Threshold for action?
      • NEEDED:
        • Project Lead (Default owner) for each component:
          • OnRamp: Larry, Bilal
          • SD-Core: Ajay, Gabriel
          • SD-RAN: Woojoong
          • ROC: ???
          • Smart5G: Alex
    • Jenkins
      • ROC and Aether-Onramp jobs only
        • Quickstart 20.04 jobs are (mostly) DYN → Still working to make robust
      • All others stopped, or moved to Github Actions
    • GitHub:
      • QUESTION:  Should I explore building a dashboard to easily see github Action statuses?
      • Still need Publish Action for ROC
  • SD-Core Status
    • Reviewing a large PR on PFCP
    • Just 4 outstanding PRs
  • SD-RAN Status
    • Updating Go modules
    • Found gRPC not working w/ latest Go (requires design work)
  • OnRamp Status
    • Adding support for USRP (going to test today)
    • Have a plan to release scale updates; to vet with Ajay and then publish for TST
  • ROC Status
  • Iowa State bringing up Aether (ARA Testbed)
    • Integrating Open Source RAN with Aether's Core
    • Bring questions to TST call and post to #aether-onramp channel on Slack
  • Have Job Templates for JJB
    • For QuickStart and 2Server jobs
    • Will open it up soon
  • Aether being used in security research

7-16-24

  • Infrastructure Updates
    • Jira → Progress stalled; hope to have a demo next week
    • Jenkins → Reduce resource usage
      • Migrate from static machines (always up) to on-demand machines
      • Clean-up across the board
      • Disabled most of the SD-Core & SD-RAN CI jobs (now done using Actions)
      • Only remaining CI jobs are for ROC
      • Expectation that we'll have things in order by mid-August
      • daf and Kunal to work together toward this (setting up a call)
    • GitHub Actions (for CI)
      • SD-Core → Done
      • SD-RAN → Done
      • ROC → Actions to Publish still todo (Sean to reach out to Gabriel)
        • Also need to grow the set of people that can modify/maintain ROC
        • Suresh has gone through the process (will send his notes to Sean, who will curate) 
        • Plan is to add a doc to to the wiki
    • Mailing Lists
      • TST public vs private
        • LF best practices is for official TST list to be limited to TST members
        • We still have the TST slack channel for "community" discussions
  • SD-Core Updates
    • Canonical has issued some substantial PRs
    • Currently being reviewed and merged
  • SD-RAN Updates
    • Woojoong could not join today
    • Currently, we are upgrading gRPC version for dependabot PRs
    • It requires onos-api update to use latest gRPC / Protobuf compiler
    • University of Glasgow looking to contribute and coming up to speed
  • OnRamp Updates
    • Looking into adding an OAI blueprint
    • Looking for opportunities to add our blueprints to SLICES 5G effort
    • Plan to publish "Research" Helm Charts for scaling the Core CP
      • Different images for now
      • Will integrate changes into source over time
      • Bilal writing a planning document
  • JJB Updates
    • Job creation with single JJB definition - Done
    • Job creation with separate yaml and pipeline definition - Done
    • Strategy is to do this for one job; will need to template to support multiple jobs

7-9-24

  • SD-Core
    • PCF code merged. 
    • Outstanding PRs from Canonical 
      • Webconsole 
      • SMF - PCF stack change to go-pfcp open.  (Tested compatibility with BESS UPF and eUPF (eBPF based))
        • UPF adapter is not considered in the above activity. We can perhaps do it once SMF changes are final/ready. 
      • AMF - AMF without mongodb. 
      • AUSF 
    • Dependabot : PRs were merges yesterday.
    • UPF debugging 
      • Youtube worked well, but iperf was going to 0 throughput. Shankar from community was using AIAB, asked him to use onRamp.
      • Recommendation is not to use GRO to get good performance. By default it's enabled. Suresh is experimenting with GRO. 
    • UPF: minor improvements in progress. 
  • SD-RAN: No representative. 
  • QA/Validation Team - No representative 

7-2-24

  • SD-RAN update Woojoong Kim 
    • Engineering team completed CI migration from Jenkins to Github action done
    • Release update
      • In July - 1.5 release
  • Jenkins Update Kunal Kapoor 
    • New group command issue:
      • Worked with Bilal, fixed now. 
      • All the native tests (onRamp deployment tests) are fixed.
    • JJB 
      • Upgraded JJB from 3.2 to 6.0.
      • Caching issue still persist.
  • SD-Core
    • 3gpp Spec Upgrade
      • SBI interface upgrade. 
      • go-pfcp:
        • SMF using local stack
        • Use go-pfcp in SMF.   
    • WIP PRs
      • Run AMF independent of mongodb. 
      • Webconsole/north bound interface of SD-Core - Static Page. 
  • Report on strategic alignment with LF (Timon)
    • Core - agreement that sd-core becomes the de facto core for LF
      • added to the 5g super blueprint
      • technical - LF wants to see Nephio support
      • marketing effort - packaging/doc/presentation/website
    • RAN - RIC evaluation project

      • Draft SOW:  

      • assembling a review committee

6-25-24 (Topics for This Meeting)

  • Aether Jira Proposal:  Aether Jira Proposal
    • Tweaked proposal and accepted as "good enough to get started"
    • Should be ready to play with next week
  • Github org renaming: https://docs.github.com/en/organizations/managing-organization-settings/renaming-an-organization
    • More representative → Mostly focused on "opennetworkinglab" (should be "aether")
    • May break external links, but now is a good time to change since we're creating new docs/pages
    • No objection...
  • Slack vs Element (https://element.io/)
    • Limiting history to 1 year (affects "grandfathered" workspaces like onf-community)
    • Proposal: prune onf-community back to "aether" and accept history limit (take up with voltha and p4)
  • Code of Conduct ((https://lfprojects.org/policies/code-of-conduct/)
    • Look through the LF standard and decide at a future meeting
  • Jenkins Status
    • ONOS GitHub Actions
      • Actions that run for PR (ROC done)
      • Actions that run for PR (SD-RAN done)
      • Not yet publishing new Docker images (in progress)
    • OnRamp Integration tests
      • JJB Related: Jobs with same name as the one created earlier, are not getting created again.
      • Native Tests: All the Jenkins jobs are not able to go beyond aether-5gc-install step. → "new group" command hanging
  • omec-pod-init and omec-cni repos Gabriel Arrobo 
    • These repos are in Gerrit → Why? Not updated in years. Move to omec GitHub? → Yes.
  • TST Meetings in July → Will remain on the schedule and expect them to happen (with some people missing)

6-18-24

  • DCO (instead of CLA) in place Larry Peterson David Ferguson (Deactivated) 
  • Configuring new Jira Larry Peterson  David Ferguson (Deactivated)
    • Archiving old Jira
    • Proposal
      • Projects: SD-Core, SD-RAN, ROC, OnRamp, Smart5G
      • Workflow: New, Assigned, Resolved, Reopened (start simple)
      • What Fields: Description, Status
      • Hopefully will demo at next week's TST
  • GitHub default branch should be main instead of master. Ajay Lotan Thakur 
    • Master is old way; main is the new way (and the default)
    • Prefer to have all the same; most currently master
    • Decision: leave it as is (expect to be mixed)
  • BESS-UPF Helm Charts: Current charts assume symmetry for the access and core interfaces (i.e., same CNI Plug-in for both) Gabriel Arrobo
    • Any objection about making the Helm Charts as generic as possible such that each individual CNI can be independently chosen?
    • Example: access=macvlan, core=host-device
    • But it remains the default that they are the same (currently macvlan); so remains backward compatible
    • Will eventually want to expose in OnRamp
  • Road map cleanup Ajay Lotan Thakur Gabriel Arrobo 
    • Priority to take up only SBI upgrade this quarter/end of Sept 2024
    • Priority on bug fixes and issues reported by other community members
    • See Roadmap for more details
    • Work harder to get info to the community / work to attract contributors
  • SD-RAN update Woojoong Kim 
    • Working with a team of 5 students
    • Github action → Almost done (run tests and publish images)
    • Scalable RAN Simulator → on going; hopefully done in Jun.
    • Dependency update → dependabot has been busy; resolving 1-by-1 (waiting on GitHub Actions)
  • OnRamp Update Bilal Saleem 
    • Separate Management Node → Dealing with local charts was the complication
    • Dueling Docker installs → Ansible playbook now checks for existing Docker, per Docker best practices
    • Check that multiple gNBs work for data plane
    • Working on proposal for incorporating scalable Core into OnRamp
  • QA Update Kunal Kapoor 
    • JJB code commit is done
    • Working on updating Readme and streamlining the code as per the current requirement.
    • David Ferguson (Deactivated) :  Customizing the job as opposed to customizing the build machine (so can dynamically grab and release VMs)

6-11-24

  • LF ticketing systems (see above)
  • CLA issues: User does not receive confirmation email to complete process (https://github.com/omec-project/pfcpsim/pull/84#issuecomment-2113633448). Additionally, CLA message points to a link that does not exist anymore (https://wiki.opennetworking.org/x/BgCUI) [Gabriel,Woojoong]
    • Woojoong: I think before it is fixed, I prefer to disable it till it's ready for SD-RAN/ONOS project.
    • Ticket filed with LF.
  • SD-Core repos: Standardize default branch name ("master" or "main") [Gabriel]
    • Lets standardize on master
  • SD-Core/Aether roadmap
    • Captured process - comments welcome
    • Current list of feature wishlist captured in roadmap - comments welcome
    • See SD-Core Roadmap work items
  • Jira access - any place its documented?
    • Should document and link in the Wiki
    • What do we document
      • Still have ONF Jira (at least in name); have migrated hosting VM to LF
      • SD-Core and SD-RAN are two projects under that Jira
      • No obvious legacy value
      • Proposal is to start fresh with Jira
  • OnRamp Improvements [Bilal]
    • Support for separate Mgmt server
    • Conflict if Docker already installed
    • Propose a blueprint for scalable Core
  • JJB Status
    • Jenkins job creation/update is now handled in the seed job
    • To document the process in README of aether-jenkins repo

6-4-24

  • Archive "tiny" (unused) repos [Gabriel]
    • As part of release 1.4, several repos were consolidated into the "util" repo
  • Do a patch release for Docker images (1.4.1) [Gabriel]
    • New Helm Chart also just published
    • Expect a PR to update OnRamp
  • Working on a blueprint for a DPDK configuration
  • SD-Core Roadmap Discussion
    • What's the process
      • We have an old Google Doc
      • Move it in the Wiki
      • Community contributes
    • How do we move past Release 15?
      • Start with OpenAPI spec for R17
      • Generate code (and test/fix)
      • Results in the wish-list
  • SD-RAN Status
    • Training new contributors
      • To document in a new wiki page for future onboarding
    • Jenkins to Github action conversion for SD-RAN is mostly done
      • Expect to complete this week 
  • ROC Status
    • PR now merged
    • New Helm Chart published by GitHub Action
    • New version (2.1.36) was published => it did not exist (but has been used by OnRamp)
    • What gives???
  • OnRamp Status
    • Bilal looking into two issues
    • Docker issue when deploying in CloudLab
    • Local helm charts on master node
  • JJB Status
    •  Testing on AWS (Goal is to be re-producible)
    • Automate Jenkins job creation using JJB and definitions from git repo
      • Seed Job on builtin node to pull the latest changes in aether-jenkins repo - Done.
      • Cron job to update the jobs in Jenkins. 
  • ...

5-28-24

  • Monitoring for Aether Registry Failures
    • Must renew certs every 3 months
    • Auto-renew often isn't so automatic (needs manual intervention)
    • Logs can trigger Slack messages → to an LF channel and #aether-dev
    • Double check on the 3-month requirement
    • Double check on having multiple eyes on the LF channel
  • Celestic Use Case (Introductions)
    • Working with MosoLab on the RAN side
    • Small server → Silicom (upper range Intel Atom); need to limit power
    • Mongo DB can't start because no AVX (current issue)
      • For 4G, should be able to go back to Mongo v4.4.6
      • Can override with local Helm Chart
    • Customers are NGOs
    • Starting with LTE (lower cost); will move
  • ROC Status → Waiting on secrets for the GitHub Actions
  • SD-RAN Status → Merged PRs to improve Actions
  • Demo JJB test case → Shoot for one e2e working example
    • Need a "test Jenkins" server
    • Deliverable: Anyone can clone aether-jenkins repo and reproduce the test (document in the repo's README)
  • QA Status
    • Multi-UPF tests are failing as the second UPF is getting crashed → in discussion with Gabriel.
      • When >1 UPF, we are getting latest UPF version (rather than default)
    • UERANSIM throughput test: In progress → Networking issue (setting up tunnel)
      • Test both TCP and UPD traffic
      • Test "output" should show the recorded numbers (on Console)
    • Exploring aether-ci-management repo to automate the new onramp integration test JJBs
      • See above.
  • OnRamp w/ local Charts
    • OnRamp runs on a different node than Aether master node
    • Bilal uses this config too, but does not local helm charts → Need the combo of both

5-21-24

  •  Draft Wordpress web site now exists (copied from previous prototype)
    • Should go live today
    • Need to point DNS at new site
    • Waiting for credentials to edit
  • Default parameters for SD-Core in OnRamp (such as use of latest version of sdcore charts, enable SCTPLB and enableDBStore) [Gabriel]
    • Latest version of Charts → Always explicitly identify latest, but manually track next version once stable
    • SCTPLB → Yes
    • enable DBStore → Yes
  • ROC Update
    • Migrating helm charts to GitHub (with Actions)
    • Waiting for daf to set up appropriate secrets
  • SD-RAN status update
    • NOTE: Sorry but Woojoong could not join TST in May as he is working from Korea - time zone is a way different. Will update the status TST meeting note in May
    • Trying to merge PRs raised by dependbot - security and functional test, which is a part of release process
    • Start investigating GitHub action to migrate it from Jenkins - try to migration done in this release cycle
    • Planning to introduce the new RAN simulator design to new members so that they can start writing a code on it in this month
    • Overall training sessions are ongoing for the new engineering team members - on track!
      • Done: How to develop sd-ran microserice
      • Done: How to use Aether OnRamp for sd-ran dev
      • Done: Golang / K8s / Helm chart basics
      • Done: Dependency among SD-RAN microservices - important to merge changes
      • Next week: K8s operator pattern / webhook service
      • Next week: Essentials for the new RAN simulator implementation
    • On-site training session(s) is scheduled next week (1-2 days)
  • QA
    • JJB files for an onramp integration test are committed on Aether-Jenkins repo.
    • Updated the default config files for UERANSIM to map it with the UERANSIM's format.
    • Debugging on UPF stuck issue with Gabriel.
      • 4G fix was easy: json → jsonc
      • 5G problem is intermittent 
    • UERANSIM throughput test: There is a change in ip address allocation, working on incorporating it.

5-14-24

  • New public calendar with all Aether meetings: https://zoom-lfx.platform.linuxfoundation.org/meetings/aether-fund
    • Update WIki/Web to point to new calendar
    • Prototype web page migrated to WordPress; live soon.
  • QA
    • Making VM-prep explicit (daf)
    • Facing an intermittent issue where UPF is getting stuck in terminating state.
      • Kunal to post link to log files to #sdcore-dev
    • Github actions for onramp integration tests.
      • Not on the immediate todo list.
      • Short-term goal use JJB defs so job config is visible & editable in GitHub
    • Throughput test with UERANSIM discussion
      • Goal is to show best practices in how to measure throughput; don't really care about the AWS numbers, per se.
  • Have moved ROC Helm Charts from Gerrit to GitHub
    • Draft GitHub Actions in place
    • Need help from daf with secrets
  • Presentation on PTF: line-rate tests of UPF

5-7-24

  • RIC Forum SMaRT-5G Demo (Presented end of March)
  • Progress migrating JJB jobs to GitHub
    • Blocker: Reproducing workable VM (starting from "clean" VM)
      • Mark Beierl has had issues with cloud-init in ci, might help
    • Blocker: Setting up "structure" on GitHub → Start with one example and find next blocker
  • Progress migrating to GitHub Actions
    • SD-Core → Fully migrated Jenkins CI jobs to Actions
    • SD-RAN → First step completed (whatever "make test" did); but "publishing" is still-to-do
    • ROC → 5 PRs open to replicated "first step"; need some help from ROC maintainers
  • Progress with Aether web site
    • Should have HTTP redirect up soon (today?)
    • Planning a new/permanent site with LF's help
  • QA
    • Debugging ngsetup failure with UERANSIM ( single server configuration) - in progress
      • 2-server config works and is tested by Jenkins integration test 
      • One of the solutions to this issue can be adding another virtual interface using Macvlan for gNodeB.
    • Working with internal IT team to get the VPN connection with AWS back up.

4-30-24 

Meeting ended early ( ~ 13:10 EDT)

4-23-24

  • Plans for next week's ONE Summit
    • Larry & Timon attending
  • SD-Core Status Update
    • Created a new tags for a source (for Canonical)
    • Only 2 open PRs (related to UPF)
    • gnbsim doc (including pointers); single source for gbnsim docs
  • SD-RAN Status Update
    • Woojoong unable to attend; nothing to report
  • OnRamp Status Update
    • Improving support for developer workflow
    • Merged PR to update to 1.0.1 SD-Core charts (maintenance release)
  • QA
    • Improvement in UERANSIM deployment like single target and code for removal of existing UERANSIM repo
      • When cloning, give it a specific name (perhaps with date) so when you delete, you delete that copy.
    • Move job definitions to GitHub - in progress
      • Created aether-onramp jobs definition for 6 jobs
      • Sync with DAF about moving from Gerrit to GitHub
  • DAF action needed to support GItHub Actions (reminder)
  • Aether web site should go live soon
    • Waiting for LF to update DNS
    • Make sure we have "open source 5G" (and similar keywords) for SEO purposes
    • Get SEO help
    • staging site (feedback from community encouraged)

4-16-24

  • LF Transition Status
    • Transferring DNS to LF has caused disruptions; so far easily correctly
    • Announcement to old ONF mailing lists to move to new mailing lists?
    • Looking into "mailchimp" like facility to do announcements (no LFID required)
    • Need a place for new people (info@opennetworking.org) to ask 
    • info@aetherproject.org works, and goes to larry, michelle, Timon
      • need to put this on the public website
    • project website needed
      • LF will link
      • Larry prototyped a site (using Larry's personal webbly account).  Need to launch.
        • TODO - get DNS to point to this site
  • SD-RAN update
    • Three grad. students from Korea will work as an engineering team member and 2-3 more will come
      • They are interested with the Scalable RAN Simulator as they are trying to do the similar thing in their lab
      • They have an experience on RICs not only SD-RAN but also Flex and O-RAN SC RIC too
      • They also want to commit their research results to the upstream as a SD-RAN xApp/rApp
    • Onboarding plan for new people
      • Woojoong has a plan to visit Korea in May. Will meet them F2F and do some hand-on session so that they can immediately start working for SD-RAN
    • Next release is planned for June
  • OnRamp Status
    • Now uses v1.4 of SD-Core (candidate branch now merged into master)
    • Were seeing some instability with integration tests on Jenkins
      • Now seem to be fixed
      • Stale info was causing problems
    • Several clean-up activities underway
      • Eliminated references to registry.opennetworking.org
      • Removing unnecessary helm repos
      • ...
    • Will make 2.2.0 official when we are ready to publicly announce
    • How to find the right slack channel → aether-onramp vs aether-dev ??
      • Update channel headers → Tell people the kinds of questions to ask (and where else to go)
  • SD-Core & OnRamp announcements
    • Tied to above bullet about Aether web site.
  • UPF, DNN, Device Group Question
    • Current open issue: https://github.com/omec-project/smf/issues/227
    • DNN is being used as the primary key in SMF
    • It is possible at the moment to represent an invalid state
      • What direction should be taken to ensure data integrity / How should this be prevented?
    • Ajay looking at issue. Canonical will help fix.
  • QA
    • All the integration test jobs are fixed 
    • Single server UERANSIM blueprint - in progress
      • 2-server config test is successful.
      • Interest in reporting throughput test results (around 800Mbps)
      • UERANSIM blueprint documentation needs to "match" what Jenkins tests
    • Next big task: move job definitions to GitHub

4-09-24

  • Status of SD-Core /  OnRamp release
    • Version 1.0.0 of SD-Core Helm Charts published (called v1.4 of SD-Core)
    • Candidate branch of OnRamp 2.2.0 passing integration tests (plan to merge later today)
    • Release Announcement in progress
  • Registries: opennetworking.org is a mirror, but is referenced by OnRamp (to be fixed before "official" release)
    • Should use aetherproject.org registry
  • SD-RAN Update
    • Recruiting PhD students to work on SD-RAN
    • Sync'ing up with Gabriel an use of GitHub actions (waiting on DAF)
    • Reminder: Take a look at SD-RAN wiki pages
  • Is it time to form Working Groups?
    • Perhaps: SD-Core, SD-RAN, Trial Deployments (or Use Cases)
  • Proposal: Move Aether-related docs to GitHub pages (freeing up AWS server that is currently being used)
    • Minimally, need to move sd-core/sd-ran docs to aetherproject.org (currently opennetworking.org)
    • Bring gnbsim docs to the set of Aether docs
  • https://wiki.onosproject.org/ is down again?

4-02-24

  • SD-Core / OnRamp release status
    • New images, Helm Charts, docs published
      • Automated through GitHub actions
      • No CI dependencies on Jenkins (SD-Core is done)
      • Also applying to other components (see below)
    • Candidate branch of OnRamp now testing
      • QuickStart blueprint passes integration test
      • Other blueprints still being tested
    • Announcement Plans
      • Obvious new channels → email, slack 
      • Can use ONF channels, but should be pushing people to the new channels
      • Timon and Larry to work up a broader plan
  • Roadmap for SD-RAN & upcoming v1.5 release
    • (See Wiki for more info)
    • Roadmap page and v1.5 release pages (proposal stage) now on Wiki
    • High-level goals
      • Catch up with latest Service Models, etc.
      • Grow the SD-RAN community
        • Recruiting PhD students
        • Possibly engineers with companies contributing to SMaRT-5G
      • Stablize new RANSIM
    • SD-RAN Release Process → Moving from Jenkins to GitHub Actions
      • Testing, creating images, unit tests, check license done
      • What's missing is the publishing-related actions
      • Also many backlogged issues
  • CI shift (Jenkins → Actions) ROC is in progress → Same done for ROC, except PRs are still pending (Jenkins issue)
  • Action for inactive PRs and issues in repositories (Gabriel)
    • https://docs.github.com/en/actions/managing-issues-and-pull-requests/closing-inactive-issues
    • https://fvsch.com/stale-bots
    • Ongoing vs One-time??
      • 140 total, but only 15 are related to 5gc
      • Other 125 issues are related to 4gc
    • Discussion: Interest in maintaining 4G variant of SD-Core
      • Don't archive it (people do use it)
      • No one actively contributing
      • 125 open issues (plus other bug fixes not yet submitted)
      • Add "we are not actively maintaining 4g" comment and close the issues
      • NOT archiving, so someone can pick it up and run
      • Still works as an OnRamp blueprint (using existing images)
      • SD-RAN still works with 4G (using existing images)
    • What about stale-bots for 5gc?
      • Currently there for 30 days; increase for 120 days
  • Registry locations (aetherproject, DockerHub, opennetworking) (Gabriel)
    • opennetworking registry seems to mirror DockerHub
    • Do we need to still publish images in opennetworking? What is the future for the opennetworking domain?
    • Wean ourselves off of opennetworking (both the registry and the DNS name)
    • General Note: Look for opennetworking.org usage and remove (check Docs!)
  • Status of UERANSIM Blueprint / QA Status
    • Changes in aether-ueransim repo - PR Raised
    • Changes in aether-onramp repo for the UERANSIM integration - PR Raised
    • Larry to review

3-26-24

  • Aether Mailing Lists now up-and-running
  • Status of SD-Core Release
    • Source code is ready for release
    • Automating tagging process (waiting on the ONF bot)
    • Replicating the VERSION approach used elsewhere in Aether → AMF is "demo" repo
    • Immediate goal is to "fix" Jenkins job to publish Helm Charts from GitHub rather than Gerrit
    • Long-term goal is to write a GitHub Action to auto-publish Helm Charts (replacing Jenkins)
    • With Respect to OnRamp 
      • Test "candidate" that uses the latest charts → Define a "candidate blueprint" and run nightly integration tests against it.
      • Upgrade OnRamp from 2.2.0-dev to 2.2.0 (previous was 2.1.0).
    • Announcement?
      • Message to announce mailing list (also Slack)
      • SD-Core Docs include Release Notes
  • Status of SD-RAN Release
  • Status of UERANSIM Blueprint / QA Status
    • 2-Server configuration is the expected default
      • 1-Server config is still problematic – data loop
      • To debug as Future Work
    • Changes in aether-ueransim repo - Done
    • Changes in aether-onramp repo for the UERANSIM integration - in progress
    • Nightly integration test added to Jenkins - in progress
  • A limitation of OnRamp is that running two Cores (related to how network is configured) can get confused. Include Bilal in a discussion of how to address. 

3-19-24

3-12-24

  • LF Transition Update
    • Wiki → Live at wiki.aetherproject.org
    • Mailing Lists → info@aetherproject.org set up (not a list); others (tst, dev, announce) still in progress
    • marketing:
      • Can set up Linkedin page? Done for VOLTHA. Marketing in general is addressed by the Board
      • Aether board has to decide on how to apply budget for marketing
    • Other board funding opportunities:
      • Intern projects (could be budgeted by the board); also funded internships
        • unfunded internships?  In return, certificate of completion??
  • Asking for help
    • Is this a discussion that should be started in the new email forum, or is Slack better for this type of discussion?
    • Slack “loses” history, email lists typically have searchable archives
    • Once the mailing list is established, Mark Beierl will kick off the conversation there.
  • UPF Performance Testing
    • Mark Beierl wants to look into trex and UPF performance testing
      • At one point ONF used "Packet Testing Framework" to stress test UPF (t-rex generates load)
      • Able to directly test UPF without any other components affecting the performance results (Intel Labs doing this manually)
      • Traffic Generator Intel Lab uses: https://github.com/gab-arrobo/upf/tree/traffic-generator
  • Request slot for presentation in 2024-03-19 TST meeting for Canonical’s Charmed Aether 5G Core (Mark Beierl and Serdar Vural)
    • Shooting for 20 minutes next week.
  • SD-RAN update (Woojoong)
    • Release plan -for v1.4
      • Targeting Q2
      • Woojoong is writing a plan (patterned after the plan for SD-Core)
      • Defining scope 
        • Major change is to use O-RAN Service Models
        • Hope to include first version of new scalable RAN-Sim
        • Alex notes the current ASN.1 encoder might not be O-RAN compliant
        • See OAI integration note below
    • Scalable RAN-Simulator - getting delayed
    • 5G OAI integration?
      •  Last Pronto demo:  OAI integrated with KPI xApp. (OAI 2022-w52)
        • needs to be released from Intel internal github repositories.  Would like to get in 1.4 if possible, but may be delayed
      • Intel Labs has separate OAI integration (OAI 2023-w27)
      • The Intel Lab changes should be merged into OAI official release; may be able to use that in 1.4 release of SD-RAN
  • QA

3-5-24

  • LF Transition
    • Wiki → Pending
    • CLA
    • AWS
    • Next Steps:
      • Define groups (group.io)
      • Invite people on old lists to join
      • Put auto-responders on old lists (waiting on wiki)
    • dev@aetherproject.org → developers across components (open for anyone to join, free to add subgroups as usage dictates, but opt for a common channel for now)
      • Anyone can post
      • Anyone can join
      • OLD lists: (e.g., omec-dev) to join this one.
    • tst@aetherproject.org → community topics / agenda / Open and public 
      • Anyone can post
      • Anyone can join
    • announce@aetherproject.org → broadcast announcements 
      • OLD email:   aether-announce (239 members). Will try to migrate
      • Event announcements, releases, milestones.  Subscribe if you want to say up on the project (expecting small volume of posts)
      • Anyone can join
      • Limit posters to TST, LF, invited people.
    • support@aetherproject.org → ask for help/info, inquiries 
      • Anyone can post
      • Small list will receive (by anyone can volunteer to receive just by joining)
    • Status
    • Set up project mailing lists; proposal:
  • Use Cases / Trial Deployments
    • bubbleRAN (OAI commercialized stack)
    • Purdue → Modified SD-Core to demo scalability of CP
    • Princeton → Demo’ing how to monitor RAN activity
    • Intel Labs → Security & Slicing (that it used OAI is an artifact that can ber feedback into a blueprint for others to benefit from
    • Larry will set up skeleton for this on wiki (once wiki is up)
    • Others will flesh out the details.
    • SMaRT-5G → Power, deployed at WINLAB
    • ESNET → Science field trial in Colorado
    • IN-MAC → Indy, Purdue/IN initiative
    • Research Platform → Purdue, Princeton, Intel Labs,...
    • Kajeet?  4G deployment
    • GSLabs +  monogoto
    • Next Step
  • Release Plans
    • In good shape; whittling away at PRs.
    • More than 200 PRs open (e.g., dependency updates)
      • Intel Labs has proposed some features (<10 PRs)
    • Have v1.5 release that was ready at one point.
    • Activity around RANSIM that’s related
    • Who!!!!!!
    • Action: Talk to Woojoong and see if he has a proposal
    • Questions: Archive SD-RAN RIC in favor of OSC RIC  (or Flex RIC)
      • Would need to fold OSC RIC back into Aether 
      • Flex RIC (OAI’s RIC) - but license might be a problem
      • SD-RAN RIC has advantages: Safe PL, cloud native design
      • Bit-rot will eventually make SD-RAN useless
    • Implies a set of “official” blueprints (which implies a set of Helm Chart
    • SD-Core
    • SD-RAN
    • Aether 2.2.0
  • QA Update
    • Few test cases are under testing phase
    • Currently local (need to move to public Jenkins)
    • Proposal : deploy a non-containerized UERANSIM on node2
    • Seems to be working ok (not tested on a single node, co-located with Core)
    • To create a Jenkins job to test (and demo)
    • gnbsim test framework 
    • UERANSIM integration

2-27-24

  • LF Transition Update (daf)
    • Impacts wiki too.
    • Certificate has been reset (this is a DNS issue)
    • Has no effect on services working
    • registry.aetherproject.org DNS resolution should be fixed sometime today.
    • AWS org had been transitioned as of Friday, but not today (looking into)
  • Guillaume to present proposal for tagging releases
    • Issue: versioning source for each NF vs versioning SD-Core in aggregate (including helm chart and container artifacts)
    • Also beyond SD-Core to include all of Aether.
  • Discussion about the process for tagging auxiliary repos (Gabriel)
    • Ghislain comment/suggestion: https://onf-community.slack.com/archives/C02TGUWTC93/p1708705492733279 
    • Yes to methodology Guillaume proposes / No need to stay in sync.
    • Historically tried to pin versions of dependencies (so builds repeatable)
    • No tag on every commit (had to change VERSION file), but the higher level module pins on a specific version, so if you want to incorporate an update, you need to change the VERSION. Developer judgment on when to update VERSION. CI mechanism does the right thing when the VERSION file changed.
  • DPDK Issues in the UPF (Mark B.) Background:
    • VM with 3 virtual interfaces, 16g ram/2x1g HP, 8 cpus
    • Setup dpdk on host for PCI ids 00:04 and 00:05 (2 of the virtual interfaces)
    • Start bessd, get error when running up4 script:
       EBUSY: Device or resource busy
    • Followup pointers from Gabriel, Dan, and Larry
  • Changing bitrate does not affect existing connections.
    • AI: Mark to open an issue.
  • QA
    • Added validations in the gnbsim integration test.
    • New framework : 33 out of 57 TCs are done (Robot-based)
    • Still seeing the issue (may be multi-NF version issue)
    • Updated webui container works better; release note has been posted; waiting for release instead of back-patching.
    • Runs in a container (apart from SD-Core/k8s)
    • Has a single interface for both control and data planes
      • Why is this a problem? A single interface works fine for others.
      • If it is a problem, is it unique to being containerized?
    • Create a more exhaustive tests for gnbsim : in progress
    • First UE failure issue
    • UERANSIM data interface connection

2-20-24

  • Attending:
    • Timon Sloane (Aether ED)
    • Gabriel Arrobo (Intel)
    • Kunal Kapoor (GS Lab)
    • Mark Beierl (Canonical)
    • Ghislain Bourgeois (Canonical)
    • Guillaume Belanger (Canonical)
    • Serder Vural (Canonical)
    • Ajay Lotan Thakur(Intel)
    • David Ferguson (LF)

Larry unable to attend today

  • CI/CD pipeline update
    • Sctplb images need to push to dockerhub/aether-registry
    • Incremental images pushed to aether registry
    • Only releases pushed to docker hub
    • PR opened but few things need to be fixed. 
    • Pending: Website publish pending
    • Pending: once github works then we can archive gerrit/opencord project for documentation
    • Github action ready
    • Docker images were pushed via jenkins job. 
    • Proposed change:
    • Documentation Job
    • Helm Chart Publication
  • QA 
    • First basic test is created here
    • Create a new framework for onramp
    • Recently seen a failure with new image as well 
      • Reported to Ajay
    • Create a more exhaustive tests for gnbsim : in progress
    • UE failures
  • AoB
    • Is there a TST next week (MWC week)?
    • General consensus is yes for the meeting
  • Release planning taking place (3:30PT Thur meeting)

2-13-24

  • Attending:
    • Ghislain Bourgeois (Canonical)
    • Timon Sloane (Aether ED)
    • Larry Peterson (Princeton)
    • Gabriel Arrobo (Intel)
    • Woojoong Kim (Intel)
    • Andy Bavier (Intel)
    • Kunal Kapoor (GS Lab)
    • David Ferguson (ONF)
    • Scott Baker (Intel)
    • Ajay Lotan Thakur(Intel)
    • Mark Beierl (Canonical)

2-6-24

  • Attending:
    • Larry Peterson (Princeton)
    • Mark Beierl (Canonical)
    • Timon Sloane (ONF)
    • Ghislain Bourgeois (Canonical)
    • Guillaume Belanger (Canonical)
    • David Ferguson (LF)
    • Ajay Lotan Thakur (Intel)
    • Gabriel Arrobo (Intel)
    • Bilal Saleem ( Purdue)
    • Andy Bavier (Intel)
    • Badhrinath Padmanabhan (Intel)
    • Kunal Kapoor (GS Lab)
  • Progress scaling SD-Core (Bilal)
  • Who are the SD-RAN contributors (approving PRs, etc) (Gabriel)
    • Woojoong → Can he help us grow this community?
    • Sarat - could he help grow this community?  PI on NTIA grant
    • Alex -- CC when responding to requests
    • Rimedo ??
  • SD-Core next release (Gabriel)
    • Define date / Target set of features/capability to be included 
    • Need to define a Roadmap
    • LF has managers and processes for releases… need to tap into
    • Support from daf
  • sdcore-helm-charts and sdcode-docs are in GitHub (Gabriel)
    • Need help to make the necessary changes in the automation pipeline to point to these repos instead of the ones in Gerrit
    • See Aether docs (related)
  • FOSSA code-scanning issues
    • Stay with approach that has token in the action
  • Progress moving aether-docs from Gerrit to GitHub (llp)
    • Replicates what use to happen on Jenkins as a GitHub Action (rather than pointing Jenkins at GitHub instead of Gerrit… the other approach)
  • Moving future TST meetings to LF tools (llp)
    • Watch slack tst channel for updates…
    • Watch this page for updates…
    • Watch for a new calendar event…
  • SD-Core wiki page down which causes the unit tests to fail in link check (issue observed while raising PR on sdcore-docs repo) (Suresh)​​ https://wiki.opennetworking.org/display/COM/SD-Core
    • ONF Wiki back up – system got corrupted, needed to be rebooted, resulted in changed IP so had to reset the DNS settings this morning.
    • Other option is to mark onf wiki as an “ignore” in conf.py (until we update docs with links to Aether wiki).
  • Aether-QA
    • Add a new integration test that “hardwires” in latest web console  image.
    • Issue is tunneling through docker container
    • Maybe look at gnbsim; how does it set up tunnel? Keeps control and user plane separate, whereas ueransim uses a single tunnel/interface.
    • Look to others that have set up UERANAIM in a separate server.
    • First ue failure → Need to update image we deploy (update helm charts in progress).
    • Iperf3 integration to ueransim
    • ROC re-deployment → No new developments to Aether-Jenkins repo on GitHub (include aether-ci-management “structure” in aether-jenkins).

Notes from earlier TST calls can be found here. Recordings can be found here.

Related pages

SD-Core Roadmap work items
SD-Core Roadmap work items
Home
Read with this
Technical Steering Team (TST)
Technical Steering Team (TST)
Home
Read with this
TST Governance
TST Governance
Home
Read with this
SD-Core
SD-Core
Home
Read with this
Research
Research
Home
Read with this
SD-RAN 1.5 Release
SD-RAN 1.5 Release
Home
Read with this