Thursday, November 9, 2017

A Short Story on “How I saved my weekend”

After a network re-design, I started realizing that the we need to change the “vlan” of all the virtual servers we created in two F5 BigIP LTM Devices…. Sadly, we have 311 Virtual Servers in each F5 and we have two devices in scope. So, the count bumped to 622… 3 clicks per virtual server and am going click 1866 times???

No... Never…

Lazy minds always find shortcuts and I started thinking about the leadership direction, automate as much you can…

Finally, after couple sighs, I started dusting my Python skills and wrote couple of lines of code to finish my playbook, named it as ‘knock-it-down.yml’ …

Yes, it took only 45-50 mins for me to save 1866 clicks and my plans during the weekend…

It was painful to accept we lost the game, but there was a smile in my face and I was trying to hide it…

Why "fw ctl zdebug..." should not be used"

Kernel debug Best Practices or "Why "fw ctl zdebug..." should not be used"

Over last several days I have seen rapidly growing amount of posts at CPUG and CP Community where "fw ctl zdebug..." command was mentioned, used and advised.

Although some of you already know my position for the matter, I have decided to write a post about the growing custom to use zdebug instead of employing full fw ctl debug mechanism.

Kernel debug in general


Check Point FW is essentially a Linux-based system with a kernel module inserted between drivers and OS IP stack. If you do not know what I am talking about, you may want to look into this post with an explanatory video for the matter.

Extracting information about kernel based security decisions is rather tricky, so Check Point developed an elaborate tool to read some info about various FW kernel modules actions.

In a nutshell, each kernel module has multiple debug flags that force code to start printing out some information. I have numerous posts in this blog explaining different flags, tips and tricks with kernel debug and also providing links to CP kernel debug documents.

Debug buffer


It is important to understand FW kernel is always printing out some debug messages. For most of the kernel modules, error and warning flags are active, and the output goes to /var/log/messages by default. This is not practical for debug, so before starting kernel debug, an engineer needs to set a buffer which would receive debug output instead of /var/log/messages file.

To do so, the following command is used: fw ctl debug -buf XXXXX, where XXXXX is the buffer size in KB. The maximum possible buffer today is 32 MB, but I advise my students to use 99999 to make sure they get maximum buffer possible anyway.

Kernel can be very chatty, so having a bigger buffer would ensure less kernel messages being lost.

Debug modules and flags


FW kernel is a complex structure. It is built with multiple modules. Each of the modules has its own flags. One can run a single debug session with multiple flags raised for several modules. To raise debug flags, one use one or several commands of this type:

fw ctl debug -m (module name) (+|-) (list of flags)

It is essential that + and - options allow you to raise and remove flags on the fly, even during an already running debug session. List of modules and flags can be found by the first link in this post.

Printing info out of buffer


Raising flags is not enough, as to get information, you need to start reading buffer out with this command:

fw ctl kdebug -f (with some options)

There will be A LOT of information, so never do this on the console. Use SSH session or redirect to a file.

Stopping debug


Once you collected the relevant info, you need to reset kernel debug to the default settings, otherwise you FW will continue printing out tons of unnecessary info. To do so, run

fw ctl debug 0

What is fw ctl zdebug then?

fw ctl zdebug is an internal R&D macros to cut corners when developing and testing new features in the sterile environment. It is equivalent to the following sequence of commands:

fw ctl debug -buf 1024
fw ctl debug (your options)
fw ctl kdebug -f
-------(waiting for Ctrl-C)
fw ctl debug 0

Why is this a problem?


If you are still reading this post and get to this line, you probably think zdebug is a god sent miracle. It simplifies so many things, it is the only way to run debug in production environment! Right? 

Wrong. To make it plain, here is the list of problematic point with this way of doing things:

1. The buffer is way too small. Lots and lots of messages might be just lost because buffers does not have enough room to hold them before read.
2. It is not flexible enough. Running debug in production requires lots of consideration and certain amount of caution. After all, you are asking FW kernel to do extra things, lots of them. The best practice is to start with a single flag or two and expand area of research in the fly trying to catch an issue. This is impossible to do with fw ctl zdebug macros.
3. It is too simple to use. You could say, what a funny argument. Yet, let's think about it. To master kernel debug as described above, one has to understand kernel structure, dependencies, flags and modules. You don't have to do any of that to run fw ctl zdebug drop, and many people do jsut that. 

And guess what, this is also the simplest way to bring your busy production FW cluster down. So no, do not try this at home or at your place of work, if job security is important for you.

Wednesday, August 21, 2013

MS Excel - Reset Comments to Original Position

Little bit of Excel knowledge is a must thing :

I ran into an issue recently, got mail with all comments not in the right position. Following code snippet helped to survive

ATL+F7 took me to vba

Pasted the following and ran this :-

Sub ResetComments()
Dim cmt As Comment
For Each cmt In ActiveSheet.Comments
cmt.Shape.Top = cmt.Parent.Top + 5
cmt.Shape.Left = _
cmt.Parent.Offset(0, 1).Left + 5
Next
End Sub

And ofcoz Review tab helped me to hide them later :)

Understanding BIOS keywords - Dmidecode

dmidecode --type {KEYWORD / Number }
You need to pass dmidecode following keywords:
  • bios
  • system
  • baseboard
  • chassis
  • processor
  • memory
  • cache
  • connector
  • slot
All DMI types you need to use with dmidecode --type {Number}:
# TypeShort Description
0BIOS
1System
2Base Board
3Chassis
4Processor
5Memory Controller
6Memory Module
7Cache
8Port Connector
9System Slots
10On Board Devices
11OEM Strings
12System Configuration Options
13BIOS Language
14Group Associations
15System Event Log
16Physical Memory Array
17Memory Device
1832-bit Memory Error
19Memory Array Mapped Address
20Memory Device Mapped Address
21Built-in Pointing Device
22Portable Battery
23System Reset
24Hardware Security
25System Power Controls
26Voltage Probe
27Cooling Device
28Temperature Probe
29Electrical Current Probe
30Out-of-band Remote Access
31Boot Integrity Services
32System Boot
3364-bit Memory Error
34Management Device
35Management Device Component
36Management Device Threshold Data
37Memory Channel
38IPMI Device
39Power Supply
Display Power supply information, enter:
# dmidecode --type 39
Display CPU information, enter:
# dmidecode --type processor
Read man page for more information:
$ man dmidecode

I see this not working in SPLAT with type switch

Monday, August 19, 2013

BIGIP F5 Command Line (bigpipe Vs tmsh)

BIGIP F5 Command Line (bigpipe Vs tmsh)

b arp show show /net arp all
b arp all delete tmsh delete /net arp all
b class DATA-GROUP mode read modify ltm data-group DATA-GROUP access-mode read-only
b class show show running-config /ltm data-group
b cluster show show /sys cluster all-properties
b config load file.ucs load /sys ucs file.ucs
b config save file.ucs save /sys ucs file.ucs
b config sync run /cm config-sync from-group/to-group DEVICEGROUPNAME
b conn show show /sys connection
b conn show all show /sys connection all-properties Show all connection table properties
b conn ss server node-ip:node-port delete delete /sys connection ss-server-addr node-ip ss-server-port node-port Delete connection table entries for node-ip node-port
b daemon list list /sys daemon-ha all-properties
b db < key name > < value > modify /sys db < key name > value < value > Modify database values
b db Platform.PowerSupplyMonitor disable tmsh modify sys db platform.powersupplymonitor value disable Disables PSU alert if only one PSU in use on Dual PSU system
b db show show running-config /sys db -hidden all-properties
b export my.config.scf save /sys scf my.config.scf v10.x only
b export my.config.scf save /sys config file my.config.scf tar-file my.config.tar v11.0+
b failover standby run /sys failover standby v11+
b fo show show /sys failover
b fo standby run /util bigpipe fo standby v10+
b ha table show /sys ha-status all-properties
b hardware baud rate modify /sys console baud-rate v10: sol10621 | v11: sol13325
b ha table show show /sys ha-status all-properties
b httpd list list /sys httpd To list httpd configuration.
b import my.config.scf load /sys scf my.config.scf v10.x only
b import my.config.scf load /sys config file my.config.scf tar-file my.config.tar v11.0+
b interface show -j show /net interface -hidden all-properties -hidden is not tab completable, but should be shown in the command output on iHealth.
b load load sys config partitions all
b merge load /sys config merge Added in v11. In v10 use bigpipe
b merge /path/to/file.txt tmsh load /sys config file /path/to/file.txt merge Merge a file into the BIG-IP configuration. Added in v11. In v10, use bigpipe
b mgmt show show running-config /sys management-ip
b monitor show show running-config /ltm monitor (?)
b nat show show /ltm nat all or list /ltm nat all-properties The two tmsh commands are required here since b nat show will list the unit preference and ARP status. Statistical information is shown via “show” while configuration information is shown via “list”.
b node all monitor show list ltm node monitor
b node show show /ltm node
b ntp servers 10.10.10.10 modify sys ntp servers add { 10.10.10.10 }
b packet filter all show show /net packet-fliter
b partition list auth partition no “show” command yet, list will only show written partitions
b persist tmsh show ltm persistence persist-records
b platform show /sys hardware
b pool list list /ltm pool
b pool show show /ltm pool members
b profile access all stats

b profile auth all show all show /ltm auth profile all The tmsh auth command does not display associated OCSP information shown by bigpipe.
b profile http ramcache show show /ltm profile http
b profile http stats show /ltm profile http
b profile ssl stats show /ltm profile ssl
b profile persist profile_name list all tmsh list ltm persistence profile_name all-properties
b profile tcp show show /ltm profile tcp
b profile tcp stats show /ltm profile tcp
b profile udp show show /ltm profile udp
b profile udp stats show /ltm profile udp
b profile xml show show /ltm profile xml
b reset load / sys default-config v10.x
b reset load / sys config default v11.x
b route show show /net route all
b rule < rule > show all show /ltm rule < rule >
b rule show show /ltm rule all
b rule stats reset reset-stats /ltm rule < rule >
b save save sys config partitions all
b self show show running-config /net self
b snat show /ltm snat
b snatpool show show /ltm snatpool
b software show sys software
b software desired install sys software image NAME volume HDX.Y reboot
b software desired install sys software image NAME create-volume volume HDX.Y v11.0+ : Creates volume and installs software. (Cannot create empty volumes in v11)
b software desired install sys software hotfix NAME volume HDX.Y Installs desired Hotfix to the specified Volume.
b stp show show running-config /net stp all-properties
b syslog list all list sys syslog all-properties
b syslog remote server none modify sys syslog remote-servers none
b syslog remote server test-srv host 192.168.206.47 modify sys syslog remote-servers add {test-srv{host 192.168.206.47}} You can append “remote-port 517″ for example to the end of the command to specify the port
b syslog remote server test-srv local ip 172.28.72.90 modify sys syslog remote-servers modify {test-srv{local-ip 172.28.72.90}} The self ip must be non-floating
b system hostname modify sys global-settings hostname NEWHOST.EXAMPLE.COM
b trunk show -j show /net trunk -hidden all
b trunk all lacp show show /net trunk detail
b unit show

b verify load load sys config verify
b version show /sys version Takes grep (but not “head” as in “b version |head”) – for example, grep on build: tmsh show sys version |grep -i build
b virtual address show show /ltm virtual-address all-properties “show” does not show the objects used by the virtual, and list does not show statistics.
b virtual all show all show /ltm virtual all-properties or list /ltm virtual all-properties “show” does not show the objects used by the virtual, and list does not show statistics.
b vlan all show all -j show /net vlan -hidden
b vlangroup all show all show /net vlan-group all
bigstart status|start|stop|restart SERVICE_NAME show|start|stop|restart sys service SERVICE_NAME
bpsh (?) load sys config from-terminal merge

Wednesday, July 3, 2013

Cisco ASA : Traffic Flow

It is always a mystery that we call ASA as a full time Enterprise Firewall. Well, may be in Paper :)

This was always been in my mind, What is the traffic flow in ASA? And it has been sometime this question haunting me,

Here is my Answer,


With VPN and Static NAT
@@@@@@@@@@


1. Check For Existing Connection (Cisco call it as ASA, LOL we know who invented it.)
2. Dest NAT
3. ACL
4. uAuth (Cut Thru proxy)
5. Source NAT
6. Encrypt (VPN)
7. RPF
8. VPN Flow
9. NAT (Host Limits : I need to put some lights here,Seems to be Connection, Embryonic limit)
10. Flow Creation

And Traffic leaves ASA

I made 4.2.2.2 as https server, just took that IP as I am familiar with that IP ;)

Evidence
@@@@

FW(config)# packet-tracer input inside tcp 172.16.1.xxx 1025 4.2.2.2 443

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
static (outside,inside) 4.2.2.2 72.163.4.161 netmask 255.255.255.255
  match ip outside host 72.163.4.161 inside any
    static translation to 4.2.2.2
    translate_hits = 0, untranslate_hits = 1
Additional Information:
NAT divert to egress interface outside
Untranslate 4.2.2.2/0 to 72.163.4.161/0 using netmask 255.255.255.255

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group acl_inside in interface inside
access-list acl_inside extended permit ip any any
Additional Information:

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: AAA
Subtype: aaa-auth
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 172.16.1.xxx 255.255.255.0
  match ip inside 172.16.1.xxx 255.255.255.0 outside any
    dynamic translation to pool 1 (60.15.22.xxx [Interface PAT])
    translate_hits = 77423, untranslate_hits = 376
Additional Information:
Dynamic translate 172.16.1.101/1024 to 60.15.22.xxx/11278 using netmask 255.255.255.255

Phase: 6
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 172.16.1.xxx 255.255.255.0
  match ip inside 172.16.1.xxx 255.255.255.0 outside any
    dynamic translation to pool 1 (60.15.22.xxx [Interface PAT])
    translate_hits = 77423, untranslate_hits = 376
Additional Information:

Phase: 7
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
static (outside,inside) 4.2.2.2 72.163.4.161 netmask 255.255.255.255
  match ip outside host 72.163.4.161 inside any
    static translation to 4.2.2.2
    translate_hits = 0, untranslate_hits = 1
Additional Information:

Phase: 9
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (outside,inside) 4.2.2.2 72.163.4.161 netmask 255.255.255.255
  match ip outside host 72.163.4.161 inside any
    static translation to 4.2.2.2
    translate_hits = 0, untranslate_hits = 1
Additional Information:

Phase: 11
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 80156, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow


VPN Without NAT (IP are Diff)
======================

`FW(config)# packet-tracer input inside tcp 172.16.1.xxx 1025 4.2.2.2 443

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group acl_inside in interface inside
access-list acl_inside extended permit ip any any
Additional Information:

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: AAA
Subtype: aaa-auth
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside) 1 172.16.1.0 255.255.255.0
  match ip inside 172.16.1.0 255.255.255.0 outside any
    dynamic translation to pool 1 (68.15.22.xxx [Interface PAT])
    translate_hits = 77311, untranslate_hits = 373
Additional Information:
Dynamic translate 172.16.1.xxx/1024 to 68.15.22.xxx/65412 using netmask 255.255.255.255

Phase: 6
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
nat (inside) 1 172.16.1.0 255.255.255.0
  match ip inside 172.16.1.0 255.255.255.0 outside any
    dynamic translation to pool 1 (68.15.22.xxx [Interface PAT])
    translate_hits = 77311, untranslate_hits = 373
Additional Information:

Phase: 7
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 80024, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow


I had this in my notes, I commented it now between 1 & 2 :)


Packet Flow Sequence
====================
PIX/ASA - Inside (Higher Sec_Lev) to Outside (Lower SEC_Level)
---------------------------------------------------------------
Eg. Type - [Sub-Type] - Description
1. FLOW-LOOKUP - [] - Check for existing connections, if none found create a new connection.
2. ROUTE-LOOKUP - [input] - Initial Checking (Reverse Path Check, etc.)

Comment : I believe DST nat should happen here so it it will match the ACL, this is proved in above example
3. ACCESS-LIST - [log] - ACL Lookup
4. CONN-SETTINGS - [] - class-map, policy-map, service-policy
5. IP-OPTIONS - [] -
6. NAT - [] - xlate
7. NAT - [host-limits] -
8. IP-OPTIONS - [] -
9. FLOW-CREATION - [] - If everything passes up until this point a connection is created.
10. ROUTE-LOOKUP - [output and adjacency]

Wednesday, June 26, 2013

How to delete IP Address from a Nokia IPSO interface when VRRP enabled on the Interface :




#Check the mode if it is monitored:

dbget -rv ipsrd:instance:default:vrrp:interface:eth-s1p3c0

#Remove interface from vrrp monitoring:

dbset ipsrd:instance:default:vrrp:interface:eth-s1p3c0:mode

#Delete the interface IP Address

clish -c "delete interface eth-s1p3c0 address 192.168.150.5"

#Save the configs

dbset save