Spencer Hastings 4512 Bimini Drive, Columbus, OH 43230
Spencer.Hastings@gmail.com • Cellular 614-402-1119 • Home 614-376-4969
Summary
As a professional with over 25 years of experience in the IT industry, I have a specialized career as a UNIX administrator supporting AIX and Linux in large (150+ server) and small environments. Including analyzing infrastructure and designing and implementing technical solutions. My background includes application and user support, systems installation, repair, maintenance, troubleshooting, problem resolution, migration, and recommendations for purchasing new equipment. I have a proven record of success at minimizing down time, reducing project time, increasing controls, and eliminating errors.
Areas of Expertise
Established and implemented many repeatable technical infrastructure requirements for security, reliability, performance, and standardization for technologies, systems, and applications.
• Design, implement, maintain, and repair of AIX and Linux environments to improve speed, reliability, and efficiency of operations and applications while meeting the demands of the business.
• Design and Implementation of data center backup systems with offsite testing of system restoration and business continuity.
• Development, implementation, and auditing of UNIX/Linux System Security Standards
• Development and maintenance of system documentation, including installation and training material.
• Scripting of multiple design solutions, throughout different areas to collect data, manipulate it, and make it useful to users and application areas.
Experience
Safelite Autoglass Corporation, Columbus, OH 2003 - Present
Sr. Infrastructure Engineer, IT (UNIX/Linux)
Provided leadership and overall vision relating to the development, implementation, and support of IT initiatives.
• Constructed complex Production, QA, and Development environments for users, administrators, and applications areas.
• Reviewed technology infrastructure for management consultant group, proposing enhancements, managing implementation projects, and assessing results.
• Proficient at establishing effective priorities among opposing requirements and setting criteria or standards.
• Have contributed to details of the DR plans and have directly participated in multiple successful “traditional” Disaster Recovery tests. I have created, planned, and implemented a "hot site" DR for AIX from 2005 to present that is tested yearly.
• Facilitated Safelite to be an IBM Test site for the "Avante" or SVC/SVCSISCO device prior to its becoming GA in Dec 2003. I personally tested this device's stability, reliability, and performance on AIX systems and gave feedback and analysis which went into the device for its GA releases, as well as into the current level of the SDD drivers for SVCCISCO on AIX, in SDD & SDDPCM.
• Contributed to the design and implementation of a large data center migration including migrations from a small number of RS/6000 servers to a large environment consisting of 150-200 virtual machines many of which are LPAR's.
• Have led technical staff in providing quality UNIX support, reliable access, regular data backup, and robust networked resources across the enterprise.
• Have supported large HA cluster environments that include traditional 2-way and 3-way clusters. Have designed and built clusters that fail over to multipurpose standby servers for minimization of wasted resources.
• Have put into play scripting solutions across the enterprise, to migrate data, and perform actions in standard ways, to mitigate “one off” solutions.
• Created paging scripting for Safelite which can do tiered group paging, with multiple groups, and interfaces. Pages can be cleared directly via phone or website.
• Have implemented and maintained IBM/HCL BES and ILMT environments allowing Linux, Windows, and AIX servers to successfully pass two IBM audits.
• Have managed Oracle licensing in which we have successfully passed 2 audits.
• Created a “CMDB” for AIX and Linux that is commonly accessed for all information about servers. Data is stored in a database, and is easily accessible via front end PHP forms. Values and objects can be selected and then dumped to the screen, or into CSV files for common use according to user desires.
• Implemented RRD tools, specifically “lpar2rrd” which contain Linux and AIX metrics, including ability to project future requirements for capacity planning.
• Built and maintained a Nagios Core environment running 4500+ service monitors over 1000 server targets of every OS. (Mainframe, AS400, Windows, AIX, Linux). Including network components such as NTP comparisons to system clocks as an example.
• Built “Golden” Images for CentOS, Redhat, and other Linux variants at Safelite. Also the scripts to “convert” a stock image into the company image for Linux servers in AWS.
• Ran a conversion project for Safelite to convert all Linux to RedHat. This converted CentOS, Gentoo, Ubuntu, Slackware, and other items to our corporate standard.
• Built Linux servers in AWS, as EC2 instances. Have used both their stock images and done conversions. I have also migrated company images to the cloud as AMI’s to use.
• Built a complete infrastructure backup for Linux, as an automated patcher. This takes snapshots of Linux boxes on VMWARE (or AMI snapshots in the cloud), patches Linux, and reboots machines within designated maintenance windows, and brings back up the applications. It automatically prunes and deletes snapshots/AMI backups a few days later. Runs against the entire company Linux infrastructure environment and makes sure that every machine is patched within a 30 day window. (Some boxes are patched weekly, depending on SLA’s and testing requirements).
• Proficient with most KEY designs, and PKI infrastructure, including certificates, openssl, openssh, jks, pgp/gpg, variants. Have constantly converted and upgraded keys over the years.
• Have built failover datacenters that use NETAPP mirroring to the other DC to stop at one DC and come up on the other.
• Currently Building DR failover for 2 distinct DC’s on premises in the IBM Cloud. Active-Passive, Active-Failover, and Active-Active, using various technologies using Oracle Dataguard and rsync, along with mksysb’s, cloud-init, and other mainstream technologies for both AIX and Linux.
AON:ISTS, Columbus, OH 1998 - 2003
Platforms Support Administrator
Design and layout of AIX based systems and support of applications to meet customer demands and management expectations for reliability, performance, and stability.
• Work with AIX 4.1-4.3 and AIX 5, maintaining the OS and hardware for approximately 50 RS/6000 servers, including multiple SP2 frames and SP switch environments in the following areas:
--Pre-installation Planning
--Installation - Hardware, Software, Network
--Management - User accounts, System resources, licenses
--Backup/Recovery
--Defining subsystems
--Performance Monitoring and Capacity planning
• TSM Administrator - Utilize ADSM/TSM versions 2, thru 5.1 to do backups, restores, archives and retrievals. Manage the TSM server, database, recovery log, storage pools, and client policies. Performed all levels of TSM administration. Set up and maintain TSM for disaster recovery including offsite and onsite storage policies and data retention.
• Perform system administration and network duties including: monitoring statuses of processes, completing backups, rebooting nodes, account maintenance, logical volume management, performance statistics, and trend analysis. Use of both PERL and SHELL scripting to create monitors and applications where necessary to escalate and resolve issues before they become problems.
• Responsible for monitoring and fixing or reporting problems with disk space, Oracle databases, Listeners, remote printing, background and foreground processing.
• Administrator/Administrative work performed for Tivoli Enterprise Console, Tivoli Service Desk, Tivoli Framework, Tivoli Storage Manager, Tivoli Distributed Monitoring, BMC Control-M, and other applications. Participated in the installation and support of 3rd party applications and their processing.
• Architect and perform maintenance on machines responsible for different levels of processing, depending upon the needs of the business, maintaining machine scalability and trending their performance management and growth. Architecting systems in line with application and ORACLE requirements to maintain database and application integrity.
• Designed and implemented AON (Columbus) Backup/Recovery system for UNIX nodes, servers, databases, and applications. A successful Disaster Recovery was performed in January 2002 and again in June 2003 from this process.
The University of Southern Colorado, Pueblo, CO 1994 - 1997
Computer Laboratory Supervisor
Managed a team of 15 assistants to keep public computer labs at the university operating at peak efficiency.
• Responsible for correcting network problems, maintaining lab operations, keeping labs supplied with necessary equipment, and keeping labs secure.
• PC duties included assisting users with Novell, DOS, Windows 3.1/95, and Macintosh systems applications and problems.
• Duties included assisting other users with UNIX (AIX and Digital) processes, commands, and administration of their accounts.
Education
The University of Southern Colorado, Pueblo, CO
B.A., English, 1997
Related Skills
• 25+ years of experience with AIX. Have worked administratively on the following levels: 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.3, 6.1, 7.1, 7.2, and 7.3
• 30 years of Linux - Have installed and administrated Slackware 2.01, 2.3, 9.1, 10.0, 11, 12, 13, and 14. Mandrake 6.0, RedHat 6, 7, 8 and 9 on Intel architecture and RedHat Enterprise Server (RHEL) 3, 4, 5, 6, 7, 8, and 9 on physical architecture or VMware virtualized environments. Have also installed and worked with RHEL on Power (ppa) architecture.
• Experienced with all manner of planning, installation, processing, performance monitoring, capacity planning, and disaster recovery planning on UNIX. I am also experienced with day-to-day user maintenance, and normal corporate workflow.
• Experienced with IBM RS/6000 and pSeries hardware including the IBM Regatta servers (P690, P670, P650, P630, and P610), SP2, and legacy hardware (H70, H80, and Microchannel based machines). I am also experience with Power Hardware (Power 5, P6, P7, P8, and P9) and their management philosophies. Experienced with dual environment VIOSERVER, and HMC based environments.
• Experienced with AIX cloning and migration mechanisms both with stand alone RS/6000 systems and LPAR configurations, including Network Installation Manager (NIM), and alt_disk_install methodologies. Movement of resources from one stack to another (LPM).
• Experienced with VG/LV Mirroring, SSA, RAID, TCP/IP, TCP Wrappers, NTP, PGP, Syslog, PSSP, SDD, microcode upgrading, Security, SSH, PuTTY/Secure CRT, Procedure documentation, and customer service and support.
• Experienced with High Availability (IBM HACMP) configurations, from both the implementation and support perspective.
• Experienced in supporting large data center environment running numerous environments. I have supported the OS side of a large numbers of different applications including but not limited to: Crossworlds, MQ Series, Siebel, Veritas Netbackup, Baan, HACMP, Oracle, DB2, Apache, TomCat, TSM, TWS, BMC Control-M, ILMT, BES, and others.
• Experienced with performance tuning and performance monitoring which include using vmstat, iostat, vmtune, io-tuning, memory tuning, ulimit tuning, and finding memory leaks within processes.
• Experienced shell scripter (bsh, bash, ksh), also familiar in perl, and php for interface work to design web forms in apache.
• Experienced with components outside of the RS/6000 arena, including the McData Fabric Switch (zoning and layout), the Storage Area Netwoking, Shark (F20 and 800 versions), and Fast-T devices. I am also somewhat familiar with CISCO SAN switches.
Specialized Training
• AIX 4.1 Basic Course Certification, IBM, Columbus, OH
Training certificate for AIX Basics course.
• Systems Administration Level I, IBM, Chicago, IL
System Administration for implementation and layout of RS/6000 systems.
• System Administration Level 2, IBM, Indianapolis, IN
System Problem Determination and Resolution on RS/6000 Systems.
• Systems Administration Level 3, IBM, Chicago, IL
Systems Administration for Performance Management and enhancement of RS/6000 systems.
• System Administration Level 4, IBM, Chicago, IL
System Administration of Storage Management for RS/6000 devices.
• RS/6000 TCP/IP , IBM, Chicago, IL
RS/6000 based TCP/IP configuration and management.
• Tivoli Framework, Tivoli Enterprise Console, Tivoli Distributed Monitoring, Tivoli, Los Angeles, CA
Training to design, implement, and maintain Tivoli Framework environments in corporate infrastructure.
• Security Ambassador, AON, Columbus, OH
Training in security concepts and key points.
• Red Hat System Administration I
Honors
• Shark Upgrade Award, AON, Columbus, OH
Award for meritous work in supporting the IBM ESS upgrades and change from IBM Brocade switches to McData.