System Journal for new CWS (Red Hat Enterprise Linux 6.4) Virtual host Hardware Summary: Storage volume map: 2013-05-09 WP Updated /etc/issue* and /etc/motd: Moved the legal notice to motd from issue (and fixed typos). hard-linked issue.net to issue. New issue: Welcome to the HCC Community Web Server (CWS) Unauthorized use is prohibited. New motd: Welcome to the HCC Community Web Server (CWS) This is an official computer system and is the property of Hillsborough Community College. It is for authorized users only. Unauthorized users are prohibited. Users (Authorized or Unauthorized) have no explicit expectation of privacy. Any or all users of this system may be subject to one or more of the following actions: Interception, Monitoring, Recording, Auditing, Inspection and Disclosing to security personnel and law enforcement officials, as well as authorized officials of other agencies, both domestic and foreign. By using this system, the user consents to these actions. Unauthorized or improper use of this system may result in administrative disciplinary action and civil penalties. By accessing this system you indicate your awareness of and consent to these terms and conditions of use. Discontinue access immediately if you do not agree to the conditions stated in this notice. No updates made yet; system isn't registered for RH support yet. Added wpollock to wheel group, enabled sudo for wheel group members. Added email alias for root to wpollock, in /etc/aliases; ran newaliases. 2013-05-10 WP Added three new repos: epel-release, rpmfusion-free-release, and rpmfusion-nonfree-release. Updated denyhosts, which saved the config file as ...rpmsave but didn't replace it. I moved that config file back (/etc/denyhosts/*.cfg) and restarted denyhosts. Install Postgres server, mostly as a test. 2013-06-05 WP Set SE Linux mode to "permissive" (from "enforcing"). Noted today I can't ssh into this host from outside the DMZ. Likely an HCC firewall issue. 2013-06-15 WP reboot sudo yum -y install procmail sudo yum -y groupinstall "Development tools" "Internet Applications" \ "MySQL Database server" "Print Server" "Printing client" \ "Security Tools" "System Management" wget ftp://ftp.gnu.org/gnu/mailutils/mailutils-2.2.tar.xz tar -tvf mailutils-2.2.tar; tar -xf mailutils-2.2.tar cd mailutils-2.2; ./configure; make; sudo make install scp yborstudent.hccfl.edu:man/man1/frm.1.gz ~/man/man1/ sudo cp ~/man/man1/frm.1.gz /usr/local/share/man/man1/ sudo yum -y install logwatch sudo rpm -e kernel-2.6.32-358.6.1.el6.x86_64 \ kernel-devel-2.6.32-358.11.1.el6.x86_64 2013-06-17 WP Edited /etc/rkhunter.conf: set "permit root login" option to no, to match ssh config. 2013-08-22 WP Changed root password. Installed some stuff (some of it for the second time): yum -y groupinstall identity-management-server internet-applications \ security-tools system-management es Updated ~root/.bashrc and .vimrc. Renamed this journal to journal-new-cws.txt 2014-03-28 WP Tried to renew dummy PKI cert using genkey; didn't work since the dummy cert ("localhost.crt") is in PEM format and genkey can't handle that. Next tried /etc/pki/tls/certs/renew-dummy-cert script, failed since original cert didn't include the private key (it was in ../private). This script clobbered localhost.crt. Lastly, tried the make-dummy-cert. That worked, and included the private key in the PEM file. Next time, the renew script should work, however, I'm not sure any service (httpd) can use the same file for the key and the cert. I guess we'll see. 2014-04-05 WP Installed unison240, jwhois 2014-07-22 WP Worked on server setup: I've been working on rebuilding this server to be the new CWS. This is difficult and time consuming, since the old CWS was an ancient Fedora 7 system and this one is RHEL 6. In addition, changes to the configuration of every service are needed, to secure the servers and to update to the new configuration formats. Finally, I don't know which hosted websites are still needed. I'm starting with configuring IPv4, HTTP, and DNS. Initially, only wpollock.com will be hosted; as others complain, I will enable their websites. For the moment, email is being left in a default configuration; I know some of the sites used out-going email (i.e. rabaut.com), but we'll see if those sites are needed. I have moved wpollock.com from /home to /var/www/wpollock.com. I feel this is a superior layout than trying to keep sites and home directories in one place. After creating those directories, I copied (via scp) html/ and secure-html/ from the old CWS. Next I updated httpd.conf and conf.d/wpollock.conf, to update references of "home" to "var/www": # httpd -S VirtualHost configuration: 169.139.223.253:80 wpollock.com (/etc/httpd/conf.d/wpollock.com.conf:3) 169.139.223.253:443 wpollock.com (/etc/httpd/conf.d/wpollock.com.conf:35) wildcard NameVirtualHosts and _default_ servers: _default_:8443 cws.hcc-online.com (/etc/httpd/conf.d/nss.conf:84) _default_:443 cws.hcc-online.com (/etc/httpd/conf.d/ssl.conf:74) *:80 is a NameVirtualHost default server www.hcc-online.com (/etc/httpd/conf/httpd.conf:1012) port 80 namevhost www.hcc-online.com (/etc/httpd/conf/httpd.conf:1012) Syntax OK However, SSL errors prevent httpd from starting. (The wpollock.com key and cert seem fine, but the dummy one for localhost causes the server to abort. Worse, wpollock.com will need to renew the certificate for the new IP anyway; that will take some time.) This can't be fixed until the server's name is restored, it's IP address updated, and the certificates and keys re-generated. *sigh* As an interim fix, I have disabled ssl.conf and removed the wpollock.com:443 site (by editing conf.d/wpollock.com, and saving that and the ssl.conf as ...-OFF.) Now that it is working, I ran chkconfig httpd on. Next, it was time to try DNS and IP configurations. First thing I noted was both NetworkManager and network services were enabled. I have disabled NetworkManager. Next, I updated /etc/hosts with the name data: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 169.139.223.22 cws.hcc-online.com cws www mail ftp 169.139.223.2 ns1.hcc-online.com ns1 169.139.223.3 ns2.hcc-online.com ns2 169.139.223.241 lcc.hcc-online.com lcc 169.139.223.242 www.hccglobal.info 169.139.223.243 www.yborfilmfestival.com 169.139.223.244 helpdesk.wpollock.com support.wpollock.com 169.139.223.245 www.floridahistory.org 169.139.223.246 www.reefsponge.com 169.139.223.247 www.towboat.org 169.139.223.248 www.brandonlaw.com 169.139.223.249 www.tampahispanic.org 169.139.223.250 www.ibello.com 169.139.223.251 www.hawkradio.com 169.139.223.252 www.schatzow.com 169.139.223.253 wpollock.com www.wpollock.com 169.139.223.254 rabaut.com www.rabaut.com (This will need updating as soon as the new IP addresses are known!) Since the server will only use static network configuration, I edited resolv.conf: search hcc-online.com hccfl.edu nameserver 0.0.0.0 nameserver 169.139.223.15 nameserver 169.139.223.4 (Those default nameservers will need updating!) Aside: It will be easier to change all the addresses using find and sed later. For now, using the assigned IP is required for network connectivity! The iptables firewall rules will need a review, once IP addresses have changed. For now, I left this at the default: *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT Next, I copied over the existing DNS zone files. named will run in a chroot jail, as before. All the zone files appear correct, although with the old IPs. I have changed the TTL for wpollock.com, to 12 hours (from 1 day). I plan to change it back after the cut-over. This should shorten the time for the Internet to start using the new data, without overwhelming the DNS server. (Update: Ken said to use 15 minutes next time.) I will add the new reverse zone, and see about having HCC delegate those (using CNAME records) to my little subnet. The named.conf file required many changes, for security configuration (that wasn't possible using the old BIND version). The new named.conf file is copied into the chroot; on the old CWS, I left a symlink in /etc/named.conf, but I prefer backups of /etc to also backup named.conf, so it is worth the extra work to maintain both copies. The commands used to check the config and zone files were: cd /var/named/chroot/var/named/ named-checkconf -z -t /var/named/chroot/ for db in db.[a-z]*; do named-checkzone ${db#db.} $db; done For reference, here is named.conf and db.wpollock.com: # cat /etc/named.conf // /etc/named.conf - This file contains the BIND version 8.x // (and later) named configuration information. // Generated by Wayne Pollock, 3-7-02 // Last changed by WP 2014-07-22: Updated for Bind 9, added security and // logging options { directory "/var/named"; /* * If there is a firewall between you and nameservers you want * to talk to, you might need to uncomment the query-source * directive below. Previous versions of BIND always asked * questions using port 53, but BIND 8.1 uses an unprivileged * port by default. */ // query-source address * port 53; version "DNS Server"; // Hide real version info. notify no; // Security setting with only one server. allow-transfer { none; }; // Disallow all zone transfers. allow-query { any; }; allow-recursion { localhost; }; // Only allow recursive queries from myself. allow-query-cache { localhost; }; // ibid. zone-statistics yes; }; controls {inet 127.0.0.1 allow {127.0.0.1; }; }; logging { channel default_log { syslog local2; severity debug; // Send all log data to syslog facility local2. }; channel security_log { syslog local2; severity debug; // Send all log data to syslog facility local2. }; category default { default_log; }; category general { default_log; }; category security { security_log; }; category config { default_log; }; category resolver { security_log; }; category xfer-in { security_log; }; category xfer-out { security_log; }; category notify { security_log; }; category client { security_log; }; category network { security_log; }; category update { security_log; }; category queries { security_log; }; category lame-servers { null; }; }; zone "223.139.169.in-addr.arpa" { type master; file "db.223.139.169"; allow-update { none; }; }; zone "61.35.192.in-addr.arpa" { type master; file "db.61.35.192"; allow-update { none; }; }; zone "hawkradio.com" { type master; file "db.hawkradio.com"; allow-update { none; }; }; zone "ibello.com" { type master; file "db.ibello.com"; allow-update { none; }; }; zone "hcc-online.com" { type master; file "db.hcc-online.com"; allow-update { none; }; }; /* zone "hawkeyenews.net" { type master; file "db.hawkeyenews.net"; allow-update { none; }; }; */ zone "hccautotech.com" { type master; file "db.hccautotech.com"; allow-update { none; }; }; zone "schatzow.com" { type master; file "db.schatzow.com"; allow-update { none; }; }; /* zone "towboat.org" { type master; file "db.towboat.org"; allow-update { none; }; }; */ zone "reefsponge.com" { type master; file "db.reefsponge.com"; allow-update { none; }; }; zone "floridahistory.org" { type master; file "db.floridahistory.org"; allow-update { none; }; }; zone "tampabaymodela.com" { type master; file "db.tampabaymodela.com"; allow-update { none; }; }; zone "rabaut.com" { type master; file "db.rabaut.com"; allow-update { none; }; }; zone "wpollock.com" { type master; file "db.wpollock.com"; allow-update { none; }; }; zone "yborfilmfestival.com" { type master; file "db.yborfilmfestival.com"; allow-update { none; }; }; zone "hccglobal.info" { type master; file "db.hccglobal.info"; allow-update { none; }; }; //include "/etc/rndc.key"; include "/etc/named.rfc1912.zones"; include "/etc/named.root.key"; =================================================== # cat /var/named/chroot/var/named/db.wpollock.com ; Name to address DNS file for primary nameserver: hcc-online.com ; Written by Wayne Pollock $TTL 43200 $ORIGIN wpollock.com. @ IN SOA ns1.hcc-online.com. wpollock.cws.hcc-online.com. ( 2014072200 ; serial 360000 ; refresh, seconds 7200 ; retry, seconds 3600000 ; expire, seconds 360000 ) ; minimum, seconds ; Nameserver(s): IN NS ns1.hcc-online.com. IN NS ns2.hcc-online.com. ; Mail record, so "user@xyz.com" is redirected to "user@mail.xyz.com": ; (Don't forget to also configure the sendmail.cf file!) IN MX 10 cws.hcc-online.com. IN TXT "v=spf1 a mx ptr -all" ; Records for this host (the primary nameserver): IN A 169.139.223.253 ; Aliases (these names are used by default by many services): www IN CNAME @ mail IN CNAME @ ftp IN CNAME @ blogs IN CNAME @ ; For HTTPS you can't use named virtual hosts so an IP is needed: helpdesk IN A 169.139.223.244 support IN CNAME helpdesk I have done my best to learn quickly the new BIND 9 security features, and apply them here: version "DNS Server"; // Hide real version info. notify no; // Security setting with only one server. allow-transfer { none; }; // Disallow all zone transfers. allow-query { localhost; }; // Only allow recursive queries from myself. allow-query-cache { localhost; }; // ibid. I also configured logging of everything to use syslog local2 facility. I plan on reviewing this with Ken (HCC's security expert) and to follow his advice, if offered. The new log file is /var/log/named.log (I didn't configure a second log for security messages). I have modified logrotate to include that file: # diff rsyslog.conf-orig rsyslog.conf 42c42 < *.info;mail.none;authpriv.none;cron.none /var/log/messages --- > *.info;mail.none;authpriv.none;cron.none;local2.none /var/log/messages 62a63,65 > # Save Bind 9 ("named") messages to its own file: > local2.info /var/log/named.log > ========================================================== # cat /etc/logrotate.d/named /var/named/data/named.run /var/log/named.log { missingok create 0644 named named postrotate /sbin/service named reload 2> /dev/null > /dev/null || true endscript } Tried to start named. Failed: permission denied for named.conf. I forgot to chgrp the file to group named. I did that in /etc and also in /var/named/chroot/etc/named.* Tried to start named. Worked, except for localhost domain! Sure enough, I needed to chgrp named the zone files too. Tried to restart named. Worked, only all logs were going to messages. I had reloaded rsyslog; now I tried to restart it instead. That fixed everything! Here's some testing: $ host www.hccfl.edu www.hccfl.edu is an alias for hccnovusweb2.hccfl.edu. hccnovusweb2.hccfl.edu has address 169.139.223.174 $ host wpollock.com localhost Using domain server: Name: localhost Address: 127.0.0.1#53 Aliases: wpollock.com has address 169.139.223.253 wpollock.com mail is handled by 10 cws.hcc-online.com. wpollock@hcc-cws /home/wpollock $ host wpollock.com wpollock.com has address 169.139.223.253 wpollock.com mail is handled by 10 cws.hcc-online.com. So named is running locally and resolving recursive queries. The only message in the log I wanted to review with Ken was this: /etc/named.conf:15: using specific query-source port suppresses port randomization and can be insecure. At this point, httpd and named (and the resolver) are working. Except of course for the incorrect IPs and SSL. I think the server is ready for a cut-over tomorrow. 2014-07-23 WP The server's IP addresses have been cut-over, and httpd is running. named has not yet been configured, and there is some SSL issue with wpollock.com. Steps taken so far: Updated kcompress password, and added to group wheel; that provides sudo ability. Changed the default runlevel from 5 to 3 in /etc/inittab. (From what I can see, RHEL uses a mix of Upstart and SysV init systems.) Asked for remote console access. Ken said okay, but I don't have it yet. It uses something called "DMS Manager", and apparently needs the VPN to use. (I have installed the new VPN client for HCC, from https://vpn.hccfl.edu/) I have determined that I can use the newer HCC backup system for YborStudent; the CWS shouldn't need it. That will have to wait for awhile. Determined the new IPs and gateway and DNS. Here's /etc/hosts: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #169.139.223.1 fwacad.hccfl.edu fwacad #10.35.61.1 gateway #10.35.61.220 yborstudent.hccfl.edu yborstudent 10.35.61.203 cws.hcc-online.com cws www mail ftp 10.35.61.204 www.hccglobal.info 10.35.61.205 www.yborfilmfestival.com 10.35.61.206 helpdesk.wpollock.com support.wpollock.com 10.35.61.207 www.floridahistory.org 10.35.61.208 www.reefsponge.com 10.35.61.209 www.tampabaymodela.com 10.35.61.210 www.hccautotech.com 10.35.61.211 www.tampahispanic.org 10.35.61.212 www.ibello.com 10.35.61.213 www.hawkradio.com 10.35.61.214 www.brandonlaw.com 10.35.61.215 wpollock.com www.wpollock.com ftp.wpollock.com mail.wpollock.com blogs.wpollock.com 10.35.61.216 hawkeye.hcc-online.com hawkeye 10.35.61.217 rabaut.com www.rabaut.com 10.35.61.218 ns1.hcc-online.com ns1 10.35.61.219 ns2.hcc-online.com ns2 # Omni Data Protector Backup Server (Tape Library): #169.139.223.54 hccbackup.family.hccfl.edu hccbackup #169.139.222.40 hcc44a.hccfl.edu hcc44a #169.139.223.78 hccbackupii.family.hccfl.edu hccbackupii #10.35.61.204 lcc.hcc-online.com lcc #10.35.61.214 www.schatzow.com #10.35.61.214 www.towboat.org Next edited /etc/resolv.conf: search hcc-online.com hccfl.edu nameserver 0.0.0.0 nameserver 10.35.61.35 nameserver 8.8.8.8 (Eventually, HCC will have a second nameserver.) The new default gateway is 10.35.61.1. The subnet mask is /24. Discussed with Ken Compres about not using 0.0.0.0. The problem is that with the new HCC network architecture, NAT is done even for the DMZ! Since /etc/hosts has the correct data, that should be good enough and using the local name server shouldn't be needed. (If we tried to use it, it would give the external IPs of 192.35.61.X, not the internal ones of 10.35.61.X.) If this change is made to resolv.conf, I will also edit named.conf to completely disallow recursive queries, currently allowed from localhost. Turned off denyhosts: it keeps adding the gateway to /etc/hosts.deny. Will hopefully figure out how to white-list some ssh IPs from that, someday. I noted that ntpd is off; it should be on but needs configuration first. TODO Added holes for ports 80 and 443 to the firewall; it is still a minimal (mostly the default) firewall for now. I learned that Ken followed my lead and blocked China from all of HCC! Anyway, the current firewall rules arei (TODO: restore complete firewall): *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT On this system, /etc/sysconfig/network is short: NETWORKING=yes HOSTNAME=cws.hcc-online.com Next, I set /etc/sysconfig/network-scripts/ifcfg-eth0 to the new info: DEVICE=eth0 TYPE=Ethernet UUID=8080dc0d-72ce-4ceb-a65f-fc7e58af147b ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=none IPADDR=10.35.61.203 PREFIX=24 GATEWAY=10.35.61.1 DNS1=10.35.61.35 DNS2=8.8.8.8 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="System eth0" HWADDR=00:50:56:8A:35:40 LAST_CONNECT=1366909244 Then I added a single range file, ifcfg-eth0-range0: # IPADDR_START -- ipaddr to start range at. eg "192.168.30.1" # IPADDR_END -- ipaddr to end range at. eg "192.168.30.254" # CLONENUM_START -- interface clone number to start using for this # range. # eg "0" IPADDR_START="10.35.61.204" IPADDR_END="10.35.61.219" CLONENUM_START="1" NETMASK=255.255.255.0 ONBOOT=yes Using a single range was made possible by shuffling some IP assignments. In the future, Ken will set HCC's reverse zone file to delegate those IPs to this server, making this (finally!) authoritative for its records. (Note: Range files have been supported in RH and Fedora since the 1990s, but never documented. Ken said it is not an issue to continue to use them.) After rebooting, the new IPs are up and working! However, there was a problem with one of them, and the default gateway. I had saved the original ifcfg-eth0 as ifcfg-eth0-orig, thinking it would be ignored by the network service. Wrong! After removing that file, restarting network service fixed everything. TODO: set rsyslog to forward logs to the central loghost. Ken told me the IP address to use, but I can't find it now. Fixed the SSL issue with wpollock. For some reason, not all the files from the old CWS transferred to here correctly. This includes several wpollock[,.-]com* files! I guess I transferred from YborStudent to here, not from the old CWS to here. That means I need to sync YborStudent with the old CWS before it is decommissioned. After replacing the certificate file in /etc/pki/tls/certs with the correct one, and restarting httpd, all is well. Updated /etc/resolv.conf: the IP I had was incorrect (an external DNS server). The new version is: search hcc-online.com hccfl.edu #nameserver 0.0.0.0 nameserver 10.140.2.64 nameserver 10.140.2.66 #nameserver 10.35.61.35 nameserver 8.8.8.8 2014-07-30 WP I found some issues with named on the CWS: The Red Hat package installed some directories with owner root. Named needs read and write access to those directories. I chown named.named on those. Next, the bind-chroot package install some directories and files in the wrong location! I moved /var/named/* to /var/named/chroot/var/named/*, and put symlinks in the original locations: root@cws /var/named # ls -l total 4 drwxr-x---. 6 root named 4096 Jul 16 17:59 chroot/ lrwxrwxrwx. 1 root root 32 Jul 30 13:44 data -> /var/named/chroot/var/named/data/ lrwxrwxrwx. 1 root root 35 Jul 30 13:45 dynamic -> /var/named/chroot/var/named/dynamic/ lrwxrwxrwx. 1 root root 36 Jul 30 13:46 named.ca -> /var/named/chroot/var/named/named.ca lrwxrwxrwx. 1 root root 39 Jul 30 13:47 named.empty -> /var/named/chroot/var/named/named.empty lrwxrwxrwx. 1 root root 43 Jul 30 13:48 named.localhost -> /var/named/chroot/var/named/named.localhost lrwxrwxrwx. 1 root root 42 Jul 30 13:48 named.loopback -> /var/named/chroot/var/named/named.loopback lrwxrwxrwx. 1 root root 35 Jul 30 13:45 slaves -> /var/named/chroot/var/named/slaves/ Next, I found that Red Hat rate-limits log messages by default, so the error messages from named never made it into the log files! That made knowing about the named misconfiguration much harder. (I'm beginning to miss Fedora. :-) I have disabled rsyslog rate-limits, restarted rsyslog, then restarted named with the new configuration. All errors appear gone now: root@cws /var/named # nslookup www.hawkradio.com 127.0.0.1 Server: 127.0.0.1 Address: 127.0.0.1#53 Name: www.hawkradio.com Address: 192.35.61.213 root@cws /var/named # nslookup www.hawkradio.com Server: 10.140.2.64 Address: 10.140.2.64#53 Non-authoritative answer: Name: www.hawkradio.com Address: 169.139.223.251