### OPEN SOURCE LICENSE - GNU AFFERO PUBLIC LICENSE Version 3.0 ####### # # Net::FullAuto - Distributed Workload Automation Software # Copyright © 2000-2018 Brian M. Kelly # # This program is free software: you can redistribute it and/or # modify it under the terms of the GNU Affero General Public License # as published by the Free Software Foundation, either version 3 of # the License, or any later version. # # This program is distributed in the hope that it will be useful, # but **WITHOUT ANY WARRANTY**; without even the implied warranty # of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Affero General Public License for more details. # # You should have received a copy of the GNU Affero General Public # License along with this program. If not, see: # . # ####################################################################### -------------------------------------------------------------------------------- NAME Net::FullAuto - Perl Based Secure Distributed Computing Network Process Automation Utility -------------------------------------------------------------------------------- NOTE TO USERS This is the most recent BETA RELEASE of Net::FullAuto. I have attemped to provide just enough documentation so that users and testers can "hopefully" get it up and running for VERY basic operations. Your help in this effort is NEEDED and will be GREATLY APPRECIATED. Please contact me at my email address - Brian.Kelly@FullAuto.com and let me know of ANY and ALL bugs, issues, problems, questions as well as suggestions for improvements to both the documentation and module itself. I will make every effort to get back to you quickly. Update the module from CPAN *often* - as I anticipate adding documentation and fixing bugs and making improvements almost daily for the immediate future. THANKS - and GOOD LUCK with your Net::FullAuto project! Brian Kelly, February 7, 2011 -------------------------------------------------------------------------------- BETA Notice WARNING: THIS IS A ***BETA*** RELEASE OF Net::FullAuto. Beware that it is provided "as is", and comes with absolutely no warranty of any kind, either express or implied. If you use the contents of this distribution, you do so at your own risk, and you agree to free the author(s) of any consequences arising from such use, either intended or otherwise. -------------------------------------------------------------------------------- SYNOPSIS use Net::FullAuto; see METHODS section below -------------------------------------------------------------------------------- DESCRIPTION Net::FullAuto is a Perl based Secure Distributed Computing Network Process Automation Utility. It's a MOUTHFUL - and it IS everything implied in it's description. Net::FullAuto is a command environent based implementation that truly embodies the term ``The Network **IS** the Computer!!'' Net::FullAuto utilizes ssh and sftp (can also use telnet and ftp, though for security reasons, this is NOT recommended) to bring the command enviroments of any number of remote computers (OS of remote computer does not matter), together in **ONE** convenient scripting space. With Net::FullAuto, you write code once, on one computer, and have it execute on multiple computers simultaneously, in an interactive dynamic fashion, AS IF the many computers were truly ONE. Net::FullAuto is POWERFUL. Net::FullAuto can be run by a user in a Menu driven, interactive mode (using the Term::Menus module - also written by Brian Kelly), OR via UNIX or Windows/Cygwin cron in a fully automated (and secure) fashion. Example: A user needs to pull data from a database, put it in text file, zip and encrypt it, and then transfer that file to another computer on the other side of the world via the internet - in ONE step, and in a SECURE fashion. Net::FullAuto is the ANSWER! Assume Net::FullAuto is installed on computer one, the database is on computer two, and the remote computer in China is computer three. When the user types the script using FullAuto, FullAuto will connect via ssh AND sftp (simultaneously) to computer two, and via sftp to computer three. Using an sql command utility on computer two, data can be extracted and piped to a text file on computer two. Then, FullAuto will run a command for a zip utility over ssh on computer two to zip the file. Next (assume the encryption software is on computer one) FullAuto will transfer this file to computer one, where it can be encrypted with licensed encryption software, and then finally, the encrypted file can be transferred to computer three via sftp. Email and pager software can be used for automated notification as well. Example: The same process above needs to run at 2:00am unattended. No Problem! FullAuto can be run via cron to perform the same actions above without user involvement. FullAuto is RELIABLE and FAULT TOLERANT. Each individual command run on a remote computer returns to FullAuto BOTH stdout (output) and stderr (error messages). With this feature, users and programmers can write code to essentially trap remote errors ``locally'' and respond with a host of error recovery approaches. Everything from sending an e-mail, to re-running the command, to switching remote computers and much more is available as error handling options. The only limits are the skills and ingenuity of the programmers and administrators using FullAuto. If FullAuto loses a connection to a remote host, automatic attempts will be made to re-connect seemlessly - with errors reported when the specified number of attempts fail. FullAuto is EASY. FullAuto uses a mix of traditional and object-oriented features to maximize ease of use and maintenance. Due to the unique nature of distributed computing, combined with the need for ease of maintaining a lot of configuration information (i.e. ip addresses, host names, login ID's, passwords, etc), along with any number of *routines* or *processes*, as well as the need for robust security, FullAuto has a unique layout and architechture. Normally in perl, programmers segregate functional code in separate script files or perl modules or packages. FullAuto supports this as well, but advocates keeping *process* code confined to a single routine in a kind of ``process library'' file. This is in order that FullAuto can provide additional built-in features like a command-handle to the local machine without having to explicitly create it. Or, being able to connect to a remote host with syntax as simple as: $computer_one=connect_ssh('COMPUTER_ONE'); IT REALLY IS THAT EASY! Commands also are easy: ($stdout,$stderr,$exitcode)=$computer_one->cmd('ls -l'); And NO CLEANUP is necessary - FullAuto handles this AUTOMATICALLY. This is a COMPLETE *routine* or *process*: sub ls_one { my ($computer_one,$stdout,$stderr,$exitcode); # Scope Variables $computer_one=connect_ssh('COMPUTER_ONE'); # Connect to Remote Host ($stdout,$stderr,$exitcode)=$computer_one->cmd('ls -l'); # Run Command if ($stderr) { # Check Results print "We Have and ERROR! : $stderr\n"; } else { print "Output of ls command from Computer One:\n\n$stdout\n\n"; } } # DONE!!AGAIN - IT REALLY IS THAT EASY! As with most things in life, what many or most consider a blessing, others consider a curse. Perl's motto is ``There's more than one way to do it.'' (TIMTOWTDI) Not everyone thinks this is utopia. Perl also attempts ``to make easy tasks easy and difficult tasks possible.'' FullAuto - written in perl - *IS* PERL. It is essentially a perl extension and therefore adheres to the same goals as perl itself: i.e. - there's no ``one'' correct way to use FullAuto. FullAuto is SECURE. It uses ssh and sftp for communication accross computers, and uses powerful encryption to store passwords to remote resources. When running FullAuto, a user on the first iteration of a process will be prompted to enter a password for each and every remote resource (or even local resource, since FullAuto can and does use ssh to acquire enhanced user-rights on the local computer.) Every following iteration will then prompt for a password ONLY ONCE (or a password can even be passed in via command or method arguement) with every other needed password retrieved from an encrypted datafile which utilizes the user's main login password as the ``salt''. For added security, and enhanced user functionality, FullAuto can be installed on UNIX computers to use ``setuid''. (Windows/Cygwin does not support ``setuid'' - so this feature is not available on Windows computers. This is the ONLY Windows limitation.) With FullAuto setup to use ``setuid'', users can be configured to run complex distributed processes in a secure fashion without the permissions ACTUALLY needed by the remote (or even local) resources. On top of that, it is possible to create a process administered by numerous individuals such that NO ONE PERSON KNOWS OR HAS ACCESS TO ALL THE PASSWORDS. For example, a database administrator on a remote computer can ``loan'' his username and password to drop a table (for instance) for a process that will be run by another user remotely. During the first iteration, after the user enters her/his password, the DB can then (when prompted), enter his/her password which will then be encrypted LOCALLY with the user's password as the salt. With the encrypted datafile and perl code protected from user write (or even read) access via setuid on UNIX computers (setup and administered by yet another individual or group - such as the root user), there is no way for either the DB to discover the user's password, or the user to discover the DB's password! Even the root user of the local computer running FullAuto will not be able to discover these passwords! (When setuid is setup and used PROPERLY). This setup will allow users to run FullAuto processes WITHOUT access to the passwords controlling remote access, or for that matter, the CODE running those processes! Reasons to use this module are: You want the output of the ps -e command from a remote UNIX computer. Example: In the file "fa_hosts.pm" add the connection information for the remote computer (This will suffice for all following examples): { 'Label' => 'COMPUTER_ONE', 'IP' => '10.200.210.37', 'HostName' => 'compter_one.w2k.fullauto.com', 'Login' => 'bkelly', 'LogFile' => "/cygdrive/d/fullauto/logs/FAlog${$}d". "${FA_Core::invoked[2]}". "${FA_Core::invoked[3]}.txt", }, In the file "usr_code.pm" add the *process* subroutine code: sub ps_one { my ($computer_one,$stdout,$stderr,$exitcode); # Scope Variables $computer_one=connect_ssh('COMPUTER_ONE'); # Connect to # Remote Host via # ssh only ($stdout,$stderr,$exitcode)= $computer_one->cmd('ps -e'); # Run Command if ($stderr) { # Check Results print "We Have and ERROR! : $stderr\n"; } else { print "Output of ps -e command from Computer One:". "\n\n$stdout\n\n"; } } Run FullAuto: (Hint: the --< # >-- line are instructions and are not displayed when the program actually runs) --< 1 >- >--------------------------- fullauto.pl --usr_code ps_one --< The user sees: >------------------------------------------ STARTING FULLAUTO on Wed Jun 6 12:27:08 2007 Starting fullauto.pl . . . Running in TEST mode computer_one Login : --< 2 >--(Hint: since 'Login' was specified in fa_hosts.pm 'bkelly' appears as the default)- Password:--< 3 >- >-------------------------- --> Logging into localhost via ssh . . . Logging into computer_one.w2k.fullauto.com via ssh . . . Output of ps -e command from Computer One: PID TTY TIME CMD 1 ? 00:00:03 init 2 ? 00:00:00 migration/0 3 ? 00:00:00 ksoftirqd/0 80 ? 00:00:00 aio/0 2805 ? 00:00:08 syslogd 2820 ? 00:00:00 irqbalance 2839 ? 00:00:00 portmap 2859 ? 00:00:00 rpc.statd 2891 ? 00:00:00 rpc.idmapd 2949 ? 00:00:00 ypbind 2969 ? 00:00:45 nscd 2987 ? 00:00:01 smartd 2997 ? 00:00:00 acpid 3059 ? 00:00:00 xinetd 3072 ? 00:00:14 ntpd 3092 ? 00:00:19 sendmail 3111 ? 00:00:00 gpm 3121 ? 00:00:03 crond 3153 ? 00:00:00 xfs 3172 ? 00:00:00 atd 3188 ? 00:00:00 dbus-daemon-1 3201 ? 00:05:09 hald 3210 tty1 00:00:00 mingetty 1432 ? 00:02:34 rvd 14675 ? 00:00:00 kdbd 17052 ? 00:00:00 postmaster 24389 ? 00:00:00 chatserv_d 16463 ? 00:00:06 java 11700 ? 00:04:48 cmefx 905 ? 00:00:00 automount 563 ? 00:00:00 sshd 564 pts/30 00:00:00 bash 641 pts/30 00:00:00 ps 642 pts/30 00:00:00 sed FULLAUTO COMPLETED SUCCESSFULLY on Wed Jun 6 12:28:30 2007 You want to zip and transfer a remote file from COMPUTER_ONE to your local computer and then unzip it: In the file "usr_code.pm" add the *process* subroutine code: sub get_file_from_one { my ($computer_one,$stdout,$stderr,$exitcode); # Scope Variables $computer_one=connect_host('COMPUTER_ONE'); # Connect to # Remote Host via # ssh *and* sftp ($stdout,$stderr,$exitcode)=$computer_one->cmd( 'echo test > test.txt'); # Run Remote Command ($stdout,$stderr,$exitcode)=$computer_one->cmd( 'zip test test.txt'); # Run Remote Command if ($stderr) { # Check Results print "We Have and ERROR! : $stderr\n"; } else { print "Output of zip command from Computer One:". "\n\n$stdout\n\n"; } ($stdout,$stderr)=$computer_one->get('test.zip'); # Get the File if ($stderr) { # Check Results print "We Have and ERROR! : $stderr\n"; } else { print "Output of zip command from Computer One:". "\n\n$stdout\n\n"; } ($stdout,$stderr)=$localhost->cmd( 'unzip test.zip'); # Run Local Command } Run FullAuto: (Hint: the --< # >-- line are instructions and are not displayed when the program actually runs) --< 1 >- >--------------------------- fullauto.pl --usr_code get_file_from_one --< The user sees: >------------------------------------------ STARTING FULLAUTO on Wed Jun 6 12:27:08 2007 Starting fullauto.pl . . . Running in TEST mode computer_one Login : --< 2 >--(Hint: since 'Login' was specified in fa_hosts.pm 'bkelly' appears as the default)- Password:--< 3 >- >-------------------------- --> Logging into localhost via ssh . . . Logging into localhost via ssh . . . Logging into computer_one.w2k.fullauto.com via sftp . . . Logging into computer_one.w2k.fullauto.com via ssh . . . Output of zip command from Computer One: updating: test.txt (stored 0%) get "/tmp/test.zip" Fetching /tmp/test.zip to test.zip /tmp/test.zip 0% 0 0.0KB/s --:-- ETA /tmp/test.zip 100% 153 0.2KB/s 00:00 Output of zip command from Computer One: Fetching /tmp/test.zip to test.zip /tmp/test.zip 100% 153 0.2KB/s 00:00 -------------------------------------------------------------------------------- METHODS Create New Host Objects connect_secure - connect to remote host via ssh & sftp ($secure_host_object,$error) = connect_secure('HOSTLABEL'); All Connect Methods return a host object if connection is successful, or error message(s) in the error variable if the method is requested to return a list. Otherwise, if the method is requested to only return a scalar: $secure_host_object = connect_secure('HOSTLABEL'); Any connection errors will result in complete termination of the process. The $secure_host_object represents both ssh AND sftp connections together in ONE object. The HOSTLABEL tag is a label to an anonymous hash block defined in the file fa_hosts.pm. (See the fa_hosts section for instructions on configuring host connection information.) The important thing to understand, is that there is no other code needed to connect to remote hosts. Net::FullAuto handles all connection details, such as dynamic remote-prompt discovery, AUTOMATICALLY. No need to define or even know what the remote prompt is. This feature 'alone' is a major departure from most other scriptable remote command and file transfer utilities. THIS IS THE RECOMMENDED BEST METHOD for CONNECTING. connect_ssh - connect to remote host via ssh ($ssh_host_object,$error) = connect_ssh('HOSTLABEL'); $ssh_host_object = connect_ssh('HOSTLABEL'); This method returns an ssh connection only - any attempt to use file-transfer features with this object will throw an error. Use this method if you don't need file-transfer capability in your process. connect_sftp - connect to remote host via sftp ($sftp_host_object,$error) = connect_sftp('HOSTLABEL'); $sftp_host_object = connect_sftp('HOSTLABEL'); This method returns an sftp connection only - any attempt to use remote command-line features with this object will throw an error. Use this method if you don't need remote command-line capability in your process. connect_host - connect to remote host via ssh OR telnet and sftp OR ftp ($host_object,$error) = connect_host('HOSTLABEL'); $host_object = connect_host('HOSTLABEL'); This method is the most powerful of all the connect methods. When this method is used, it will first attempt to connect to the remote host via ssh and sftp. However, if for any reason either or both ssh and sftp fail to connect, then it will attempt to connect via telnet and/or ftp. (Use connect_reverse for the opposite behavior.) This method returns either a ssh or telnet connection and either a sftp or ftp connection. (Note: you could get a connection that is telnet/sftp or ssh/ftp) Note: This is the most powerful method, but not the most secure, becasue it's possible to connect with telnet and/or ftp. Use this when process completion is more important than having optimum connection security. connect_insecure - connect to remote host via telnet & ftp ($insecure_host_object,$error) = connect_insecure('HOSTLABEL'); All Connect Methods return a host object if connection is successful, or error message(s) in the error variable if the method is requested to return a list. Otherwise, if the method is requested to only return a scalar: $insecure_host_object = connect_insecure('HOSTLABEL'); Any connection errors will result in complete termination of the process. The $insecure_host_object represents both telnet AND ftp connections together in ONE object. The HOSTLABEL tag is a label to an anonymous hash block defined in the file fa_hosts.pm. (See the fa_hosts section for instructions on configuring host connection information.) THIS METHOD IS *NOT* RECOMMENDED for CONNECTING - use connect_secure() whenever possible. connect_telnet - connect to remote host via telnet ($ssh_host_object,$error) = connect_telnet('HOSTLABEL'); $ssh_host_object = connect_telnet('HOSTLABEL'); This method returns a telnet connection only - any attempt to use file-transfer features with this object will throw an error. Use this method if you don't need file-transfer capability in your process. THIS METHOD IS *NOT* RECOMMENDED for CONNECTING - use connect_ssh() whenever possible. connect_ftp - connect to remote host via ftp ($ftp_host_object,$error) = connect_ftp('HOSTLABEL'); $ftp_host_object = connect_ftp('HOSTLABEL'); This method returns an ftp connection only - any attempt to use remote command-line features with this object will throw an error. Use this method if you don't need remote command-line capability in your process. THIS METHOD IS *NOT* RECOMMENDED for CONNECTING - use connect_sftp() whenever possible. connect_ssh_telnet - connect to remote host via ssh OR telnet ($host_object,$error) = connect_ssh_telnet('HOSTLABEL'); $host_object = connect_ssh_telnet('HOSTLABEL'); When this method is used, it will first attempt to connect to the remote host via ssh. However, if for any reason ssh fails to connect, then it will attempt to connect via telnet. (Use connect_telnet_ssh for the opposite behavior.) This method returns either a ssh or telnet connection. Note: This is a powerful method, but not the most secure, becasue it's possible to connect with telnet. Use this when process completion is more important than having optimum connection security. This method returns a remote command-line connection only - any attempt to use file-transfer features with this object will throw an error. Use this method if you don't need file-transfer capability in your process. connect_telnet_ssh - connect to remote host via telnet OR ssh ($host_object,$error) = connect_telnet_ssh('HOSTLABEL'); $host_object = connect_telnet_ssh('HOSTLABEL'); When this method is used, it will first attempt to connect to the remote host via telnet. However, if for any reason telnet fails to connect, then it will attempt to connect via ssh. (Use connect_ssh_telnet for the opposite behavior.) This method returns either a telnet or ssh connection. Note: This is a powerful method, but not the most secure, becasue it's possible to connect with telnet. Use this when process completion is more important than having optimum connection security. Also, this method will return a telnet connection FIRST if available. Use this if connection reliability is important, but performance is more important than security. This method returns a remote command-line connection only - any attempt to use file-transfer features with this object will throw an error. Use this method if you don't need file-transfer capability in your process. THIS METHOD IS *NOT* RECOMMENDED for CONNECTING - use connect_ssh() whenever possible. connect_sftp_ftp - connect to remote host via sftp OR ftp ($host_object,$error) = connect_sftp_ftp('HOSTLABEL'); $host_object = connect_sftp_ftp('HOSTLABEL'); When this method is used, it will first attempt to connect to the remote host via sftp. However, if for any reason telnet fails to connect, then it will attempt to connect via ftp. (Use connect_ftp_sftp for the opposite behavior.) This method returns either a sftp or ftp connection. Note: This is a powerful method, but not the most secure, becasue it's possible to connect with ftp. Use this when process completion is more important than having optimum connection security. This method returns a file-transfer connection only - any attempt to use remote command-line features with this object will throw an error. Use this method if you don't need remote command-line capability in your process. connect_ftp_sftp - connect to remote host via ftp OR sftp ($host_object,$error) = connect_ftp_sftp('HOSTLABEL'); $host_object = connect_ftp_sftp('HOSTLABEL'); When this method is used, it will first attempt to connect to the remote host via ftp. However, if for any reason sftp fails to connect then it will attempt to connect via sftp. (Use connect_sftp_ftp for the opposite behavior.) This method returns either a ftp or sftp connection. Note: This is a powerful method, but not the most secure, because it's possible to connect with ftp. Use this when process completion is more important than having optimum connection security. Also, this method will return a ftp connection FIRST if available. Use this if connection reliability is important, but performance is more important than security. This method returns an file-transfer connection only - any attempt to use remote command-line features with this object will throw an error. Use this method if you don't need remote command-line capability in your process. THIS METHOD IS *NOT* RECOMMENDED for CONNECTING - use connect_sftp() whenever possible. connect_reverse - connect to remote host via telnet OR ssh and ftp OR sftp ($connect_reverse_object,$error) = connect_reverse('HOSTLABEL'); $connect_reverse_object = connect_reverse('HOSTLABEL'); When this method is used, it will first attempt to connect to the remote host via ssh and sftp. However, if for any reason either or both telnet and ftp fail to connect, then it will attempt to connect via ssh and/or sftp. (Use connect_host for the opposite behavior.) This method returns either a telnet or ssh connection and either a ftp or sftp connection. (Note: you could get a connection that is telnet/sftp or ssh/ftp) Note: This is a powerful method, but not the most secure, because it's possible to connect with telnet and/or ftp. Use this when process completion is more important than having optimum connection security. Also, this method will return a telnet/ftp connection FIRST if available. Use this if connection reliability is important, but performance is more important than security. Host Object Methods cmd - run command line commands on the remote host ($cmd_output,$error) = $connect_secure_object->cmd('hostlabel'); There is a cmd method available with every connect_object. For all objects that contain both remote command-line and file-transfer connections, the cmd method gives access ONLY to the remote command-line feature. To access the ftp cmd options, use the following syntax: ($ftp_cmd_output,$error) = $connect_secure_object->{_ftp_handle}->cmd('help'); For all objects that contain only a file-transfer connection, the cmd method gives access ONLY to the file-transfer command-line feature. ($sftp_cmd_output,$error) = $connect_sftp_object->cmd('help'); -------------------------------------------------------------------------------- EXAMPLES -------------------------------------------------------------------------------- AUTHOR Brian M. Kelly -------------------------------------------------------------------------------- COPYRIGHT Copyright (C) 2000-2018 by Brian M. Kelly This program is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License. (http://www.opensource.org/licenses/AGPL-3.0).