Network Engineer Sonic Full Time NEW

Job Title: SysEng – Network Engineer

Location : Remote but base location will be Noida office

Communication should be excellent

Position Overview

Looking for a network engineer with experience in datacenter environments and at least light

programming experience.

●SONiC programmatic iterative configuration (gnmi/yang, swss) experience required

●SONiC base configuration (L2, mclag, lag/portchannel, bgp, bfd, etc) experience preferred

●FRR experience preferred (OSPF)

●OpenGear experience nice to have

●Light systems programming language (C, C++, Golang, Rust, etc) experience preferred, stronger

experience nice to have

●Scripting language (Python, Bash, etc) experience required

●Linux administration (Bash, systemd units, general system navigation) experience preferred

●Virtual networking (VXLAN) experience preferred

●AWS networking (VPC, Direct Connect) nice to have

Task Expectations

1. Programmatic Iterative Configuration of SONiC Switches (yang/gnmi, swss, etc.)

Expected experience/abilities:

●Has used the above previously to configure, or can trivially identify how to implement CRUD

operations (or at least CRD) against constructs such as but not limited to:

○Physical and sub interfaces

○VXLAN/VNI

○VRF

○ACL

○At minimum must provide correctly functioning examples

●Functionality will ultimately be written in Golang. A network engineer is merely expected to

identify, document, and demonstrate interface functionality for the SingleStore team to

implement.

○Network engineer being able to implement the CRUD/CRD functionality as a

library/module in Golang would be a plus but is unexpected

○Actual virtual networking control plane implementation is expected to be responsibility

of SingleStore team

○Network engineer contributing here would be high value, freeing up team to focus on

storage implementation

2. Base Configuration of SONiC Switches

Inband Configuration:

●L2, mclag, lag/portchannel, bgp, bfd

●Need to set up anycast addresses for metadata service IP, SAG, etc

Bifurcated Spine-Leaf Topology (Inband):

●Each side of aisle has 2x spines to be mclag’d

●Each spine has 2x connections to each other spine to be lag’d

●Each spine has 2x connections to each ToR/leaf on same side to be lag’d

●Each side of aisle has private AS

●ToR-compute node connections breakouts, ToR-storage nodes standard

Spine-Leaf Topology (Out of Band):

●Each side of aisle has 1x spine

●Each spine has connection to each ToR/leaf on both sides

●Currently each side of aisle has private AS

○Can be argued should be single AS

●Currently hardcoded L3

Additional Notes:

●Spine model in use insufficient resources for unified DHCP stack – had to settle on model due to

tariff season (isc dhcpd usable)

●Server BMCs previously static IP’d and/or infinite lease’d via DHCP by vendor, require crash

carting/manual full reset in order to DHCP

●Switch management ports physically connected but not currently configured to be reachable via

OOB network

●PDU management ports do not DHCP, require on-site troubleshooting to bring into network

Coordination:

●Coordinate with DevOps on switch integration with monitoring

●Transition from ad-hoc to code-driven base configuration

○Coordinate with DevOps on switch provisioning (ZTP or otherwise)

○Coordinate with DevOps on SONiC build pipeline

3. Palo Alto Firewall Configuration Remediation

●Transition from ad-hoc to code-driven configuration

Multipath & Traffic Handling:

●Ensure multipath functioning correctly

○Firewall rules engine appears to favor single source interface for all src/dst resulting in

erroneous packet drops

●Ensure upstream egress/ingress A/P functioning correctly

○Will need to work with network team of colocation vendor providing IP transit to

remedy IP transit only having one functioning leg at present

○May require on-site work/coordination

Direct Connect:

●Ensure direct connect multipath correctly working

●Ensure no overly eager security features negatively impacting legitimate traffic (session

drops/throttling, unreasonable latency impacts – currently ~200ms hit on some traffic)

●Ensure no unlicensed security features enabled (dnssec currently erroneously enabled)

Additional Configuration:

●NAT public IPs for use

●Interzone traffic rules currently permissive – more mature tiered scheme necessary for long term

●Coordinate with DevOps on firewall integration with monitoring

4. Console Network Setup

Physical Topology:

●2x OpenGear OM2224 spines

●16x OpenGear IM7248 ToRs

Current State:

●Spines routable, providing loop for firewall management ports

●Cellular not active

●ToRs lack ethernet routing

●All end-device access currently through nested console sessions

Requirements:

●FRR experience required, OpenGear experience bonus

Challenges & Fixes:

●OpenGear cellular fallback does not work well with multipath (destroys routing when triggered)

○Should be manually implemented using systemd timer with heartbeats over multiple

paths

Setup Tasks:

●Set up direct-to-end-device serial console via SSH

●Configure standardized versions for IM7248s and OM2224s

●Set up standardized credentials

Cellular Requirements:

●Business cellular plan required for OM2224s

○Minimum 10GB/month, 50GB+ preferred

○Needed for emergency access and recovery scenarios

Constraints:

●AT&T allows OpenGears but blocks Palo Alto traffic on consumer plans

●Verizon 4G coverage inconsistent in colo area

●One T-Mobile 4G band unsupported by OM2224 modem

Coordination:

●Work with colocation vendor for antenna extension installation on roof to ensure reliable cellular

signa

Job Type: Permanent

Pay: ₹217,273.67 – ₹4,005,056.17 per year

Work Location: Remote