Aruba – AP Troubleshooting

In order to make the WLAN function, the access points need connectivity to the controller.  Let’s review what an AP does during the boot process

Acquire IP address (can be static or acquired from DHCP)

  • IP address is required for communication with the controller using PAPI and GRE
  • To verify the access point properly acquires an IP you will need console access to the AP

Discover Controller

  • The AP goes through the following process in trying to discover a controller
  1. Statically assigned
  2.  DHCP Vendor Option 43
  3. ADP Multicast: Group Address 239.0.82.11 (requires multicast routing to be enabled on infrastructure)
  4. ADP L2 Broadcast
  5. DNS (aruba-master.<localsuffix>
  • The AP will follow the sequence exactly as above.  Once the AP learns a controller address, it terminates the discovery process and attempts to communicate with the learned address.  If the AP doesn’t receive a response from the learned address, then the AP will initiate a full reboot and start the process again

Update Code if necessary

  • AP Compares the code level to the controller’s code level
  • If the code revision matches the AP will continue the boot process
  • If the code revision does not match the AP will obtain the new code from the controller using FTP (TFTP is used on the initial join or if the AP is purged
  • The AP will automatically reboot after the code upgrade/downgrade
  • “show ap database” shows the current status of each AP (will list if upgrading, rebooting, etc)

Obtain Configuration Information

  • Once an AP connects to a controller and has compatible code, it will receive its configuration over PAPI
  • “show ap config ap-name <ap-name>” will show the AP configuration being pushed to the AP

Build GRE Tunnel

  • GRE is used to carry all of the wireless traffic between the AP and the local controller
  • A GRE tunnel is created per SSID per AP
  • The AP System Profile Controller LMS-IP setting tells the AP which controller the AP should terminate with
  • Be sure to allow Protocol 47 between the controllers and APs
  • “show ap debug system-status ap-name <ap-name>” – shows the communication status between the controller and AP
  • “show datapath tunnel table”shows the GRE tunnels established with the controller (look for prt 47)
  • “show ap debug counters”shows how many times an AP has rebooted or bootstrapped

Enable Radio

  • Once the GRE tunnel has been established the Radios will become enabled
  • “show profile-errors” – shows the list of invalid user created profiles.  An invalid user profile could cause the AP not to broadcast its assigned SSIDs.

Add a Local Controller to an Existing Master

Once a network has scaled up beyond a single controller capacity, a master/local architecture is recommended.  This allows the master controller to handle configuration and correlation while the locals can be responsible for handling the user traffic.

Steps to Add a local controller to an existing Master:

From the Master GUI:

Configuration -> Network -> Controller -> System Settings

–          Click New Local Controller IPSec Keys

–          Enter the IP Address of the local controller that you are adding

–          Enter an IPSec key

–          Re-Enter the IPSec key

–          Click Add

–          Click Apply (don’t forget)

–          Save Configuration

Or From the Master CLI:

localip <local ip address> ipsec <ipsec key>

From the Local GUI:

Configuration -> Network -> Controller -> System Settings

–          Select Controller Role Drop-Down and select Local

–          Enter the Master Controller IP (VRRP address if using master redundancy)

–          Enter IPSec Key  (Same key you entered on Master)

–          Click Apply

–          Reboot Controller

Or From the Local CLI:

masterip <master ip address> ipsec <ipsec key>

Master Controller Redundancy – Configuration

I’m going to walk through the steps for configuring master redundancy.  Here is my scenario:

Primary Master – 10.10.10.11 (should always be the master if up)

Backup Master – 10.10.10.12

VRRP IP – 10.10.10.15

VRRP ID – 10

VLAN – 10

First let’s start with the VRRP configuration on the Primary Master.

config t

vrrp 10

vlan 10

ip address 10.10.10.15

priority 110

preempt

description Preferred-Master

tracking master-up-time 30 add 20

no shut

Now let’s configure the Backup-Master

config t

vrrp 10

vlan 10

ip address 10.10.10.15

priority 100

preempt

description Backup-Master

tracking master-up-time 30 add 20

no shut

Once VRRP is up I need to associate the VRRP instance with master controller redundancy:

On the Primary:

config t

master-redundancy

master-vrrp 10

peer-ip-address 10.10.10.12 ipsec aruba123

On the Backup:

config t

master-redundancy

master-vrrp 10

peer-ip-address 10.10.10.11 ipsec aruba123

Once the VRRP instance is associated with master redundancy then I need to synchronize the WMS and local user database between the two controllers:

config t

database synchronize period <minutes> – defines the scheduled time to sync the databases (minimum should be 20 minutes)

Controller Redundancy

For my first technical deep dive let’s get into controller redundancy. During this post I will define the different types of redundancy in the Aruba system.  Please no controller vs controller-less rants!

Let’s begin by defining redundancy.  According to Wikipedia, redundancy is the duplication of critical components or functions of a system with the intention of increasing reliability of the system.  Unfortunately redundancy is left off a lot of wireless network designs due to cost.  In today’s mobility first environments redundancy needs to be implemented properly to ensure the reliability of the mission critical WLAN.

In Aruba world we have four levels of controller redundancy:

1)      Fully redundant – includes both master and local redundancy

2)      Redundancy aggregation – local redundancy

3)      Hot Standby – Local access points fail-over to Master

4)      No Redundancy – self-explanatory (far too common)

Master Redundancy:

The first controller redundancy model we will look at is Master redundancy.  The master controller is the control plane of the centralized WLAN.  The master controller is responsible for handling the global configuration of the WLAN system, location tracking, IDS event correlation and alerting.  The first question we should ask ourselves is what happens if the master controller is unavailable?  If the master becomes unavailable all master functions are lost (configuration, location tracking, and IDS) but the WLAN itself will continue to function.  New and existing clients will still be able to access the WLAN while the master controller is down.

To provide redundancy for the master controller we will setup a master/standby relationship with two controllers.  The Standby Master is a hot standby controller.  The Standby Master will not terminate AP sessions while it is the backup unit.  Updates on the state of the network are sent from the active Master to the Backup.  The two controllers sync the databases (WMS and local user) at a configured interval (typically 30 minutes).

VRRP (Virtual Router Redundancy Protocol) is used as the redundancy mechanism between the two controllers.  VRRP requires Layer2 adjacency.  The two master controllers will use a shared VRRP interface address.  The VRRP address is used by local controllers, access points, and mobility access switches to discover the master controller on the network.  The VRRP address can also be used by network administrators to access the management interface for the current master controller.

Local Redundancy:

Next we will look at Local Controller Redundancy.  The Local Controllers in an Aruba WLAN are responsible for AP termination, user authentication, and policy enforcement.  If a local controller fails and there is no backup the WLAN will become unavailable.

Local controllers have three methods for redundancy

Active-Active

  • two locals share a set of APs,  divide the load, acts as a backup for each other
  • if the two controllers are L2 adjacent, run two instances of VRRP with each controller acting as a primary for one instance and backup for the other instance
  • if the two controllers are not L2 adjacent then you will need to setup a LMS/Backup LMS IP address in the AP System Profile
  • You can also combine VRRP and LMS/Backup LMS for a more robust redundancy design, the VRRP addresses can be used as the LMS/Backup LMS IP addresses

Active-Standby

  • similar to Active-Active except one controller sits idle while the primary controller supports the full loads of APs and users
  • this model has a larger failure domain (increases latency because the full load must failover to the backup
  • typically this model utilizes the LMS/Backup LMS configuration
  • you could also use a single VRRP instance if the controllers are L2 adjacent

Many to One

  • typically used in remote networks where branch offices have local mobility controllers but redundancy onsite is not feasible
  • a large controller is deployed as the +1 controller at the data center
  • failure typically occurs across a WAN link
  • preemption should be enabled in this scenario due to the possible delay introduced by failing over to a remote site

No Redundancy

  • if the local goes down, no users can connect
  • Any AMs associated go down

Now that we know the different types of redundancy options we need to be aware of a few rules to ensure our network stays up according to plan.  There are four major rules in dealing with controller redundancy:

  1. Make sure the redundant controller can support the additional AP load during a failover event
  2. Make sure the same VLANs exist on both controllers and that named VLANs are mapped on the redundant controller
  3. Make sure the controllers are running the same OS version
  4. Make sure the redundant controller has the same license features enable and ensure you have enough license capacity to support the additional AP load during a failover event (AOS 6.3 will address this previous limitation)

In my next post I will begin configuring each of the different redundancy methods.

SSL VPN configuration on SRX running 15.1X49-D80.4 or higher

Starting with version 15.1X49-D80.4 the Juniper SRX supports dialup vpn over a connection to port 443 with the NCP client. It needs some specific configuration to get that working and we found out the hard  way. So, we have decided to share it here.  Thank you Valentijn and Jasper for helping me.

The situation we want to achieve is this one:

To prepare for configuring a demo setup you need two things: A gateway running a Junos version that supports this feature and a NCP client. You should know how to get and install the SRX software, you can get the client here: https://www.ncp-e.com/en/resources/download-vpn-client/

The configuration we’re about to make gives us a dialup vpn where the client tries to connect to with standard IPsec. If that fails it will try to move the connection to SSL, which in many networks is allowed to travel freely…

Two profiles are configured to authenticate the user:
1)             lpdap-users: to authenticate against the AD control on 172.27.72.10, domain wsa.local

2)             local-users: In which two local users are defined.

Both profiles hand out IP addresses and DNS servers from the address assignment pool dyn-vpn-address-pool.
Please note we use rather weak proposals, just for testing purposes, in real life adjust them to your (companies) policy!

Phase 1 config
set security ike proposal my_ncp_proposals authentication-method pre-shared-keys

set security ike proposal my_ncp_proposals dh-group group2

set security ike proposal my_ncp_proposals authentication-algorithm md5

set security ike proposal my_ncp_proposals encryption-algorithm aes-128-cbc

set security ike proposal ncp-client authentication-method pre-shared-keys

set security ike proposal ncp-client dh-group group2

set security ike proposal ncp-client authentication-algorithm md5

set security ike proposal ncp-client encryption-algorithm aes-128-cbc

set security ike policy ike_ncp_client mode aggressive

set security ike policy ike_ncp_client proposals my_ncp_proposals

set security ike policy ike_ncp_client pre-shared-key ascii-text <key>

set security ike gateway ncp_test ike-policy ike_ncp_client

set security ike gateway ncp_test dynamic user-at-hostname “[email protected]

set security ike gateway ncp_test dynamic ike-user-type shared-ike-id

set security ike gateway ncp_test external-interface ge-0/0/0.0

set security ike gateway ncp_test aaa access-profile ldap-users   *

set security ike gateway ncp_test version v1-only

set security ike gateway ncp_test tcp-encap-profile ssl-vpn

* You can change this to profile local-users to authenticate the users locally instead of against LDAP.

The last line of configuration tells the device to accept TCP encapsulated traffic according the mentionedprofile. Here is how to configure that profile:

set security tcp-encap profile ssl-vpn log

Since ike and tcp encapsulated traffic will arrive at the external interface, both should be accepted as host inbound traffic:

set security zones security-zone untrust host-inbound-traffic system-services ike

set security zones security-zone untrust host-inbound-traffic system-services tcp-encap

Because we want ssl vpn traffic on the interface no other listener should be enabled on the interface: make sure system service web-management https is not enabled on the external interface. Enabling it   on that interface would be a bad idea anyway.

Let’s take a look at the authentication profiles, starting with the ldap profile:

set access profile ldap-users authentication-order ldap

set access profile ldap-users authentication-order password

set access profile ldap-users domain-name-server 172.27.72.16

set access profile ldap-users domain-name-server 172.27.72.17

set access profile ldap-users client mtepper firewall-user password “$9$.PQ30ORSyK36pB1hKv4aJ”

set access profile ldap-users address-assignment pool dyn-vpn-address-pool

set access profile ldap-users ldap-options base-distinguished-name DC=wsa,DC=local

set access profile ldap-users ldap-options search search-filter sAMAccountName=

set access profile ldap-users ldap-options search admin-search distinguished-name CN=administrator,CN=Users,DC=wsa,DC=local

set access profile ldap-users ldap-options search admin-search password “$9$Cze7uIheK87NbM8ZUDjq.uOB1SreKM”

set access profile ldap-users ldap-server 172.27.72.10 port 389

set access profile ldap-users ldap-server 172.27.72.11 port 389

As you can see the administrator account is used here for a lookup. In real life you might want to create an account with just the necessary rights in the Active Direcory domain. Also note that you need to adjust the base-distinguished-name to your own domain.

For a simple test you could use a profile with local users like this:

set access profile local-users client jverdonk firewall-user password “$9$m5nCOBESlMz3EyeW-dZUjkmTQFn/Ap”

set access profile local-users client mtepper firewall-user password “$9$xXNNbYDjqf5FYgGiHmF3cyr”

set access profile local-users address-assignment pool dyn-vpn-address-pool

Both profiles use the same address pool for address assignment configuring this pool isn’t a hard task as well:

set access address-assignment pool dyn-vpn-address-pool family inet network 192.168.3.0/24

set access address-assignment pool dyn-vpn-address-pool family inet xauth-attributes primary-dns 172.26.72.16/32

set access address-assignment pool dyn-vpn-address-pool family inet xauth-attributes secondary-dns 172.27.72.17/32

This makes the configuration complete for phase 1 and phase 1½ (meaning for Xauth, which asks for    authentication between phase 1 and phase 2). Time to look at phase 2 config then. According the documentation about SSL VPN we found a route based VPN with tunnel interface in point to point mode is needed to get things working. So, we configured this:

An interface in the security zone trust (best practice for production is creating a zone called VPN and use that to make clear what happing in your policies) and an intrazone security policy:

set interfaces st0 unit 0 family inet

set security zones security-zone trust interfaces st0.0

set security policies from-zone trust to-zone trust policy default-permit match source-address any

set security policies from-zone trust to-zone trust policy default-permit match destination-address any

set security policies from-zone trust to-zone trust policy default-permit match application any

set security policies from-zone trust to-zone trust policy default-permit then permit

 

Finally for the SRX we can configure the phase 2: (As in phase 1 in real use stronger proposols!)

set security ipsec proposal dialup-ncp protocol esp

set security ipsec proposal dialup-ncp authentication-algorithm hmac-md5-96

set security ipsec proposal dialup-ncp encryption-algorithm aes-128-cbc

set security ipsec proposal dialup-ncp lifetime-seconds 3600

set security ipsec policy ipsec_ncp perfect-forward-secrecy keys group2

set security ipsec policy ipsec_ncp proposals dialup-ncp

set security ipsec vpn Ipsec_ncp bind-interface st0.0

set security ipsec vpn Ipsec_ncp ike gateway ncp_test

set security ipsec vpn Ipsec_ncp ike ipsec-policy ipsec_ncp

set security ipsec vpn Ipsec_ncp traffic-selector test local-ip 0.0.0.0/0

set security ipsec vpn Ipsec_ncp traffic-selector test remote-ip 0.0.0.0/0

The gateway  is ready now, time to move to the client.

After installing the software, start it and go into the configuration of a profile. Configure things like shown here: any tab not shown is left default!

The relevant part of the config of the SRX should look like this:

security {
    ike {
        proposal my_ncp_proposals {
            authentication-method pre-shared-keys;
            dh-group group2;
            authentication-algorithm md5;
            encryption-algorithm aes-128-cbc;
        }
        proposal ncp-client {
            authentication-method pre-shared-keys;
            dh-group group2;
            authentication-algorithm md5;
            encryption-algorithm aes-128-cbc;
        }
        policy ike_ncp_client {
            mode aggressive;
            proposals my_ncp_proposals;
            pre-shared-key ascii-text "$9$MB7WdbUDk5T3P5M8"; ## SECRET-DATA
        }
        gateway ncp_test {
            ike-policy ike_ncp_client;
            dynamic {
                user-at-hostname "[email protected]";
                ike-user-type shared-ike-id;
            }
            external-interface ge-0/0/0.0;
            aaa {
                access-profile ldap-users;
            }
            version v1-only;
            tcp-encap-profile ssl-vpn;
        }
    }
    ipsec {
        proposal dialup-ncp {
            protocol esp;
            authentication-algorithm hmac-md5-96;
            encryption-algorithm aes-128-cbc;
            lifetime-seconds 3600;
        }
        policy ipsec_ncp {
            perfect-forward-secrecy {
                keys group2;
            }
            proposals dialup-ncp;
        }
        vpn Ipsec_ncp {
            bind-interface st0.0;
            ike {
                gateway ncp_test;
                ipsec-policy ipsec_ncp;
            }
            traffic-selector test {
                local-ip 0.0.0.0/0;
                remote-ip 0.0.0.0/0;
            }
        }
    }
    policies {
        from-zone trust to-zone trust {
            policy default-permit {
                match {
                    source-address any;
                    destination-address any;
                    application any;
                }
                then {
                    permit;
                }
            }
        }
    }
    tcp-encap {
        profile ssl-vpn {
            log;
        }
    }
    zones {
        security-zone untrust {
            host-inbound-traffic {
                system-services {
                    ike;
                    tcp-encap;
                }
            }
        }
        security-zone trust {
            interfaces {
                st0.0;
            }
        }
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            family inet {
                address 1.2.3.4/24;
            }
        }
    }
    st0 {
        unit 0 {
            family inet;
        }
    }
}

access {
    profile ldap-users {
        authentication-order [ ldap password ];
        domain-name-server {
            172.27.72.16;
            172.27.72.17;
        }
        client mtepper {
            firewall-user {
                password "$9$.PQ30ORSyK36pB1hKv4aJ"; ## SECRET-DATA
            }
        }
        address-assignment {
            pool dyn-vpn-address-pool;
        }
        ldap-options {
            base-distinguished-name DC=wsa,DC=local;
            search {
                search-filter sAMAccountName=;
                admin-search {
                    distinguished-name CN=administrator,CN=Users,DC=wsa,DC=local;
                    password "$9$Cze7uIheK87NbM8ZUDjq.uOB1SreKM"; ## SECRET-DATA
                }
            }
        }
        ldap-server {
            172.27.72.10 port 389;
            172.27.72.11 port 389;
        }
    }
    profile local-users {
        client jverdonk {
            firewall-user {
                password "$9$m5nCOBESlMz3EyeW-dZUjkmTQFn/Ap"; ## SECRET-DATA
            }
        }
        client mtepper {
            firewall-user {
                password "$9$xXNNbYDjqf5FYgGiHmF3cyr"; ## SECRET-DATA
            }
        }
        address-assignment {
            pool dyn-vpn-address-pool;
        }
    }
    address-assignment {
        pool dyn-vpn-address-pool {
            family inet {
                network 192.168.3.0/24;
                xauth-attributes {
                    primary-dns 172.26.72.16/32;
                    secondary-dns 172.27.72.17/32;
                }
            }
        }
    }
}