CTS 2311 (Unix/Linux Security) Lecture Notes

By Wayne Pollock

Lecture 1

Welcome!  Introduce course.  Review syllabus, RSS, wiki.  Your ua?? is active.

Review: Admin I and II topics: Basic Sys Admin tasks, hardware types, terminology, and management; partitioning and disk management; booting; system boot-up and shutdown; change management, help desk, administrative policies and procedures; disaster recovery; basic network configuration and trouble-shooting (ping, route); starting and stopping services; printing; managing users and groups and disk quotas; backups and archives, file locking and basic security, cron and at, source code versioning and building, open source and other licenses, the kernel and related concepts, system and service monitoring, basic networking, time servers, time zones and locales.

Discuss Project 1 (mention LVM not required) install Linux, pass out Fedora DVDs (also available from DistroWatch.com).  Discuss system journal — Use wiki.

Have students use wiki to pick host names and user IDs (need to be unique in class).  Mention post install tasks (on web).

Definition of Information Systems Security (INFOSEC):

From the NSA (U.S. National Security Agency):  Protection of information systems against unauthorized access to or modification of information, whether in storage, processing or transit, and against the denial of service to authorized users, including those measures necessary to detect, document, and counter such threats.

Qu: what do we mean by “security”?  (Short Answer: CIA)  We mean protection of valuable assets, not just from people with malicious intentions (“black hats”, or attackers) but protection from accidents including environmental disasters.  You also must consider how long you need to keep information secret; that can be anywhere from a few minutes to over 70 years.

Attackers work in seven steps: reconnaissance, weaponization, delivery, exploitation, malware installations, command/control, and exfiltration.  This model is sometimes known as the kill chain.  Lockheed-Martin defines these steps this way:

1.    Reconnaissance - Research, identification and selection of targets, often represented as crawling Internet websites such as conference proceedings and mailing lists for email addresses, social relationships, or information on specific technologies.  (This is sometimes called footprinting.)

2.    Weaponization - Coupling a remote access trojan with an exploit into a deliverable payload, typically by means of an automated tool (weaponizer).  Increasingly, client application data files such as Adobe Portable Document Format (PDF) or Microsoft Office documents serve as the weaponized deliverable.

3.    Delivery - Transmission of the weapon to the targeted environment.  The three most prevalent delivery vectors for weaponized payloads by APT (advanced persistent threat) actors (attackers), as observed by the Lockheed Martin Computer Incident Response Team (LM-CIRT) for the years 2004-2010, are email attachments, websites, and USB removable media.

4.    Exploitation - After the weapon is delivered to victim host, exploitation triggers intruders’ code.  Most often, exploitation targets an application or operating system vulnerability, but it could also more simply exploit the users themselves or leverage an operating system feature that auto-executes code.

5.    Installation - Installation of a remote access trojan or backdoor on the victim system allows the adversary to maintain persistence inside the environment.

6.    Command and Control (C2) - Typically, compromised hosts must beacon outbound to an Internet controller server to establish a C2 channel.  APT malware especially requires manual interaction rather than conduct activity automatically.  Once the C2 channel establishes, intruders have “hands on the keyboard” access inside the target environment.

7.    Actions on Objectives - Only now, after progressing through the first six phases, can intruders take actions to achieve their original objectives.  Typically, this objective is data exfiltration, which involves collecting, encrypting and extracting information from the victim environment; violations of data integrity or availability are potential objectives as well.  Alternatively, the intruders may only desire access to the initial victim box for use as a hop point to compromise additional systems and move laterally inside the network.

You can never guarantee 100% safety no matter what you do.  So there is a trade-off between how secure you make a system (how hard you make it for attackers to damage or steal your assets, how much protection you provide against power loss, floods, etc.), and how quickly you can recover from an incident, versus how much money you spend, and what sort of procedures you implement.  (Your employees and customers won’t accept just any procedure.  I doubt a strip-search and full body X-ray for every employee every day would be accepted!)

The balancing of costs and protections makes security an exercise in risk management.  First you need to determine what needs protection, and what the threats are that you plan on defending against.  This is a threat assessment (and produces a threat matrix).  Next you define security policy that, if implemented correctly, will reduce the risk of the identified threats, to an acceptable level.  The policy is implemented using various procedures and security mechanisms.

Security raises issues, not just of protection by technology, but legal, ethical, and professional issues as well.  (There is often a difference between personal and professional ethics.)  You need to worry about policies, assurance, and security design of software (sadly neglected in programming courses and books!)

For any organization, there are requirements for security.  (Here we are concerned only with Information Systems’ Security (IS security).  Security requirements can be self-imposed or can originate from an external source.  For example, companies that process any type of U.S. government information are subject to the provisions of the Federal Information Security Management Act (FISMA) and the associated National Institute of Standards and Technology (NIST) Special Publications (there are lots of these).  Companies that process national security information are subject to security requirements published by the Committee on National Security Systems (CNSS).  Other U.S. standards are Federal Information Processing Standards (FIPS) and American National Standards Institute (ANSI) standards.  Organizations that process credit card data need to comply with the payment card industry (PCI) Data Security Standard (PCI DSS).  Note that many standards organizations adopt each others’ standards, so a given standard may have an ANSI number, a FIPS number, and ISO number, and so on.

There are other requirements at the global, federal, state, or (regulatory) industry level for security of health information, corporate financial data, protection of employees and customers, etc.  Nearly all these requirements have the same general security goals: confidentiality, integrity, and availability (CIA), discussed later.

History of Information Security

Thousands of years old:

·       Kings to other rulers (secret agreements)

·       Leaders to soldiers (military communications)

·       Finance (banking, trading, merchants to partners and branches)

·       More recently, expectations of privacy:

o   Medical records (including genetic data)

o   Employment and other records

o   Mail

o   TV viewing, web surfing

o   Computer communications: email, IM, ...

o   Location (GPS and other tracking via cell phones, cars, ...)

·       Computer transactions (i.e., buying stuff on-line, auctions, ...)

Early encryption: Julius Caesar’ cipher: A->C, B->D, C->E, ...

Early bookkeeping: 8500 B.C.E.!

Double-entry bookkeeping: About 1400 C.E.  Every transaction is posted to two separate books, in one as a credit and in the other as a debit.

Example: Accounts receivable book and the cash account book: A customer pays $100 owed.  That is a debit in the Accounts receivable book (the company is now owed $100 less than before, so that account balance goes down), and a credit in the cash accounts book (the company now has $100 more than before).

The two books are maintained by separate clerks.  (A slip recording the transaction by some teller has one copy go to each clerk, which is why bank forms used to be multipart carbons).

At the end of the month/week/day (banks are daily), the owner/manager collects the books and compares the totals, which must exactly balance each other.

This system of dual control prevents any one employee from cheating the firm.

Seals and Security Printing:  In 2,000 BC Mesopotamia warehouse keepers would take small marker objects or tables known as bullae, one for each item a customer stored there, bake the bulla into a clay ball known as an envelope and make an official seal on the wet clay.  Later a customer could reclaim items by presenting the envelope intact to the warehouse keeper, who would break it open (after inspecting the seals) and allow the customer to take away each item matched by the bulla.

Seals were and are used to authenticate documents.  (Ornate seals are supposed to be hard to forge.)  For example, a signet ring would be used to make an impression in some sealing wax melted over a lock.  This wax is brittle and if the seal is broken it cannot be resealed without the original signet ring, without detection.

Today seals have evolved into security printing.  Examples include currency (the Intaglio process and others), watermarks (today, digital watermarks), and price stickers on merchandise that can’t be lifted off without ripping.

Tamper resistance means that something can’t be changed (or in some cases, examined) easily.  Tamper evident means no changing (or examination) without leaving evidence.  Examples include foodstuffs and medicine bottles, and computer cases that warn (or reset) if the case is open (HCC uses these!)  Modern versions include smart-cards.  Seals and security printing are mechanisms to provide data integrity (by being tamper evident).

Side-channel Attacks:  Why attempt to break down the front door if the backdoor is open?  Some examples include:

Emission security refers to preventing a system from being attacked using conducted or radiated signals, known as Van Eck radiation.  Examples include power analysis (power consumption monitoring): writing a “1” to an EEPROM may consume more power than writing a “0”, and analyzing RF signals given off by monitors, cables, etc. allows an attacker to determine what data is written!

In WWI (1914), field phones were used to talk to headquarters from the front lines.  These were literally grounded, but it was found the signals could be heard in other field phones hundreds of feet away!

More modern military systems can include TEMPEST hardening, to prevent emission radiation vulnerability.  This is not just radiation shielding, but power isolation and timing obfuscation.

Together, these non-direct attacks are known as side-channel attacks.  They can be extremely effective!  One such attack known as padding oracle attacks was recently (6/2012) used to crack RSA SecurID (and other secure tokens), in about 15 minutes!  This attack extracts the private keys stored on devices that use PKCS#11.

While you need defense against such side-channel attacks, you shouldn’t neglect the front door — that is continue to use proper procedures, audits, strong passwords, etc.

Modern information security involves information stored in computers, and thus its history stems from early computer security work done since 1950.  Most innovations have been discovered only in the past 30 years or so, making “InfoSec” a young discipline.

Early on, computers were not networked, but ran one batch job at a time.  No security was implemented, but having the next batch job read leftover data stored on disk or RAM was a potential problem.

By the end of the 1960s, multi-tasking computers permitted multiple users to run jobs concurrently.  One of the first information security related publications was in 1968 by Maurice Wilkes, discussing passwords.  Even today, people don’t heed that advice!

With the growth of networking computers, security became a more difficult concept to fully understand, let alone implement.  For example, Internet protocols evolved from ARPAnet, which was concerned that enemies might crash a vital computer.  The protocols invented (which later became TCP/IP) were designed to keep the network functional even if a few nodes were knocked out.  However, it was apparently assumed that all users of the network were friendlies, and all nodes and users would “play by the rules”.  Clearly this assumption is no longer true, if it ever was.

Points to keep in mind:

·       Install security updates.

·       Remove (or don’t install) unnecessary software.

·       Accept what you can and cannot secure.

·       Security management is a balancing act:  If you do it wrong users will try to find ways around it.  If they can get around it you’re not doing your job right!  If they can’t get around it they might just give up and leave for another company (or complain to your boss).

·       Keep an iron grip on access control.  Don’t let developers ever touch production servers.  Doors that should be locked must have windows so you can see who’s in the room.

Many of the security controls designed to protect information systems are directly related to one of the other security disciplines. FIPS Publication 200 establishes 17 general categories of security controls that must be applied to protect information and information systems.  They are:

·       Access control

·       Awareness and training

·       Audit and accountability

·       Certification, accreditation, and security assessments

·       Configuration management

·       Contingency planning

·       Identification and authentication

·       Incident response

·       Maintenance

·       Media protection

·       Physical and environmental protection

·       Planning

·       Personnel security

·       Risk assessment

·       System and services acquisition

·       System and communications protection

·       System and information integrity

Ethics, Laws, and Customs

Laws restrict the availability and use of various security mechanisms, and require the use of others in certain circumstances.  Laws and professional standards (legal and acceptable practices) may require (or forbid) certain security policies.  (Examples: No encryption for email in France, must state privacy policy on commercial U.S. websites, employers cannot require polygraph examinations of employees.)  Note it is legal for U.S. companies to require DNA and blood samples from all employees, but it is not socially acceptable.

Since the attacks on the U.S. on 9/11/2001, the federal government has passed a large number of laws given them authority to secretly spy on anyone.  (See, for example, CALEA, which allows for broad tapping of all communications including Internet use.)  Even when a court order is required, you may never know about the surveillance or have any opportunity to challenge it.  In a paper published in 2012, A federal judge estimates that his fellow federal judges issue a total of 30,000 secret electronic surveillance orders each year—and the number is probably growing.  Though such orders have judicial oversight, few emerge from any sort of adversarial proceeding and many are never unsealed at all.  Those innocent of any crime are unlikely to know they have ever been the target of an electronic search.

Security Organizations and Certifications

CERT (cert.org, the computer emergency response team, operates a security coordination center, CERT/CC (hosted by the Software Engineering Institute of Carnegie-Mellon Univ.).  CERT now provides a certification for security incident handling.

www.cybercrime.gov is run by the US Dept. of Justice (US-DOJ) and provides a way to report phishing and other incidents, and advice on responding to incidents.

The Internet Crime Complaint Center (IC3) is a partnership between the Federal Bureau of Investigation (FBI), the National White Collar Crime Center (NW3C), and the Bureau of Justice Assistance (BJA).  UC3 receives, develops, and refers criminal complaints of cyber crime.  The IC3 gives the victims of cyber crime a convenient and easy-to-use reporting mechanism that alerts authorities of suspected criminal or civil violations.  (www.ic3.gov)

www.securityfocus.com hosts Bugtraq, vulnerability lists, alerts, and job information.  A security professional should monitor CERT/CC and Bugtraq (or sign up to the alerting system, mailing lists, or RSS).  See also GovInfoSecurity.com, which provides news and other resources related to U.S. Government security issues (run by a private security company).

Disa.mil (and DISA Information Assurance) - The DISA is the U.S. federal agency charged with keeping all federal government computers secure.  The standards for this was the DOD Information Assurance Certification and Accreditation Process (DIACAP).  In 3/2014, the DoD announced a major change, dropping DIACAP in favor of a family of NIST standards called the RMF (risk management framework). (See DoDD 8510.01_2014.)

The DHS (Department of Homeland Security) runs many security programs (www.dhs.gov).  (The DHS, the FBI (Federal Bureau of Investigation), and the military, all play roles in computer and network security for the U.S.)

The U.S. NSA also provides security guides, under their information assurance program.  See the one for Red Hat 5.

us-cert.gov is run by the DHS, and provides regular advisories and incident reports which are required reading for all SAs (subscribe now at www.us-cert.gov/cas/techalerts).  (These are summarized in regular bulletins, with links to the National Vulnerability Database for more details including fixes; show one from a recent bulletin.)  The site is also a rich source of information about security in general.

The ENISA (European Network and Information Security Agency) is roughly the equivalent of US-CERT for the European Union.  (They don’t produce regular advisories or incident reports however.)  In 2012, the EU created CERT-EU.

US-CERT also hosts ICS-CERT, which provides information and alerts for industrial control systems, such as those hacked by Flame and Stuxnet, the 2010 worm designed to cripple Iran’s nuclear program (see 6/2012 NY Times story.  Stuxnet and Flame were made and deployed by the U.S. (and Israel) as part of a cyber-espionage (or terrorism) program known by the code name Olympic Games.  More recently (6/2012), ICS-CERT reported a similar attack on the U.S. oil pipelines.

The NICC or National Infrastructure Coordinating Center, which replaced the NIPC (National Infrastructure Protection Center), is also run by the DHS.

The FBI deals with cybersecurity (security of computers and networks) with several initiatives.  (www.fbi.gov/cyberinvest/cyberhome.htm)

SANS, the SysAdmin, Audit, Network, Security organization (sans.org) is a trusted and the largest source for information security training and certification in the world.  It maintains the largest collection of research documents about various aspects of information security, and it operates the Internet’s early warning system, the Internet Storm Center.  The certification offered by SANS, the GIAC (Global Information Assurance Certification, giac.org) is considered by many to be the toughest to get (and hence valuable).

IMPACT, the International Multilateral Partnership Against Cyber-Terrorism is an organization that aims to become a platform for international cooperation on cybersecurity.  Its advisory board features tech luminaries like Google’s Vint Cerf and Symantec CEO John Thompson.  The group’s forthcoming World Cyber Security Summit (WCSS), which will be part of the WCIT (World Congress of IT) 2008, is an effort to raise IMPACT’s profile as an international platform for responding to and containing cyberattacks.  (Quote posted on Ars Technica about this: “Must be the on-line counterpart organization to the Strategic Homeland Intervention, Enforcement, and Logistics Division.  Who’d they hire for names?  Stan Lee?”)

CISSP (Certified Information System Security Professional, cissp.com), ISSEP (Information System Security Engineering Professional) are certifications granted by International Information Systems Security Certification Consortium (ISC²) isc2.org.  Holders of these certifications must know the common body of knowledge that defines the terms and concepts professionals use to communicate, and also includes best practices (including security management), some relevant law (for the U.S. anyway), ethics, and other knowledge and skills.

EC-Council (eccouncil.org) provides “ethical hacking” training, resources, and highly regarded certifications.

There are other certifications for security (DoD-8140, security+, ...).  HCC now offers AS/AAS/CCC in this.  (Prof. Ron Leake runs this program.)  Other groups that offer certifications: iapsc.org, ipsa.org.uk, ...

NIST’s ITL (National Institute of Standards and Technology, Information Technology Laboratory) publishes security related information.  For example, see NIST-SP (special publication) 800–27, which presents a list of 33 basic principles for information technology security.  CISSPs need to know many NIST SP 800-X pubs, such as X=12 (Intro to Comp. Sec), 14, 18, 26, 27 (I.T. Security Principles), 30 (risk management).  Also OMB circular A-130 (policy for federal government agencies to purchase, manage, and secure IT systems; updated in 2015 for the first time in 15 years).  Another important standard is ISO-27002 (which replaces ISO-17799) Code of Practice for Information Security Management.

ISO-27001 is part of a family of ISO/IEC standards (8 so far in 2009) often referred to as ISO27K.  This standard deal with InfoSec Management Systems (ISMS).  It is often implemented together with ISO-27002, which lists specific security controls.  Organizations that claim to have adopted ISO/IEC 27001 can be audited and certified compliant.  (ISO/IEC 27002 provides best practice recommendations on InfoSec management for use by those who are responsible for initiating, implementing, or maintaining ISMS, within the standard in the context of the C-I-A triad.)

The National Vulnerability Database (NVD) is a repository maintained by the U.S. government of all known vulnerabilities.  This includes security related software flaws, misconfigurations, and product names and versions.  The NVD also includes an impact statement for each vulnerability (uses CVSS).  It keeps the vulnerabilitity lists for various products (Windows 7, Fedora, etc.), known as checklists, in SCAP format, allowing automated vulnerability scanning as well as compliance reporting.

The Common Vulnerabilities and Exposures (CVE) database (at Mitre.org) is the most comprehensive, internationally and freely available database of known vulnerabilities and the malware that exploits them.  It assigns a unique identifier to each.  These CVE numbers are wildly used everywhere, including SCAP, NVD, US-CERT bulletins, etc.  Every new exploit is recorded here.

Security Regulations for Credit/Payment Card Industry (PCI)

Many commercial organizations handle electronic payment information, such as by processing credit (also debit) card payments.  In the past each brand of card (Visa, Amex, ...) had security regulations for handling customer names, addresses, and other collected payment info.  (For example: an un-encrypted log file is not allowed to store a complete credit card number, but you can store the first 6 and last 4 digits.)

Merchants and payment processors need to file notices of compliance with the PCI Data Security Standard (PCI DSS), usually done by an internal audit.  Recently all major banks have agreed to a single set of standards.  The PCI Security Standards Council was founded by Amex, Discover, JCB, Visa, and MasterCard.  Each organization agreed to adopt the standards that the group decides on.  See www.PCISecurityStandards.org for more information.

See also www.linuxsecurity.com (a good site for information, but not for certification), and mybulletins.technet.microsoft.com (Security bulletins from MS, customized for your computer).

The number of payment card data breaches and fraud (stolen card data from self-service gas stations, Walmart, etc.) is huge and growing.  Currently, the customer and retailer is protected by law from liability; the bank associated with the card must pay.  One technology can help with this, smart cards, also known as chip cards.  The swipe stations that require both a chipped card and a user-entered PIN are known as chip-and-pin.

As of October 1, 2015, liability for payment card fraud will shift from card companies to retailers if the retailers have not upgraded their terminals to accept chip-based payment cards.  The cards have been used in Europe for 10 years, but the US has been slow to adopt the technology, largely due to the associated costs that merchants will have to bear.  The Target breach is what drove the industry to set a timeline for adopting the standard. — ComputerWorld

Live CDs / DVDs

There are a number of bootable Linux live CDs, that contain collections of useful security tools.  (A good list can be found at www.securitydistro.com/security-distros.)  Examples include Auditor, Phlak, Whoppix, and Pentoo, but most of these haven’t been updated in several years.  Some more recent examples include BackTrack (a merger of Auditor and Whoppix), Network Security Toolkit (NST), Helix, DEFT, and DVL (a purposely broken distro, designed for learning and training).

The Black Market for Malicious Hackers

[Adapted 10/17/07 from: en.wikipedia.org/wiki/E-gold and  www.cmu.edu/news/archive/2007/October/oct15_internetblackmarkets.shtml.]

Today there is a thriving market where people can buy stolen credit card numbers, purchase malware or spam software, even hire developers.  This market even uses a reputation system (think about e-bay or Amazon.com’s Marketplace), so buyers can tell if a seller is reliable or just going to take their money and run.  The 2007 CSI survey reported that U.S. companies on average lost more than $300,000 to cyber crooks.

“These troublesome entrepreneurs even offer tech support and free updates for their malicious creations that run the gamut from denial of service attacks designed to overwhelm Web sites and servers to data stealing Trojan viruses,” said Perrig, a professor at CMU. “...monitoring found that more than 80,000 potential credit card numbers were available through these illicit underground web economies”, said a PhD student at CMU working with Perrig.

You can easily hire hackers these days, say from the Hackers List website.

As of 2012, LilyJade malware is available on malware markets for around $1,000.  This is a web browser extension that works with IE, Firefox, Chrome, and other browsers, on any platform.  LillyJade appears to be focused on click fraud, spoofing ad modules on Yahoo, YouTube, Bing/MSN, AOL, Google and Facebook.  It also has a Facebook-based proliferation mechanism, which spams users with a “Justin Bieber in car crash” style message, complete with a link to a location where a user can be infected.  (Reported in The H 5/23/12.)

Most personal computers that get infected with malware are targeted by pay-per-install (PPI) services, which reportedly (Technology Review 6/9/11) charge up to $180 per 1,000 successful installations.  Typical installation schemes involve uploading tainted programs to public file-sharing networks, hacking legitimate Web sites to download automatically the files onto visitors’ machines, and quietly running the programs on PCs they have already compromised.

Payments on the black market

Whatever the purchases, a buyer will typically contact the black market vendor privately using email, or in some cases, a private instant message.  Money generally changes hands through non-bank payment services such as e-gold, making the criminals difficult to track.  e-gold.com (and others such as e-bullion) provide accounts backed 100% by gold or silver deposits.  These companies allow instant transfers from one account (the buyer’s) to another (the seller’s).  Since these transactions are not part of the international banking system, they are usually impossible to trace.  E-gold is also known as private currency as it is not issued by governments.

Opening an account at e-gold.com takes only a few clicks of a mouse.  Customers can use a false name if they like because no one checks.  (Some such as GoldNow are more reputable and do require security checks.)  With a credit card or wire transfer, a user can buy units of e-gold.  These units can then be transferred with a few more clicks to anyone else with an e-gold account.  For the recipient, changing e-gold back to regular money is just as convenient and often just as anonymous.

Note that e-gold doesn’t convert the deposits to any national currency.  E-gold does not sell its e-metal directly to users.  Instead digital currency exchangers such as OmniPay (a sister company of e-gold) and numerous independent companies act as market makers, buying and selling e-metal in exchange for other currencies and a transaction fee.  In this manner, e-metals can be converted back and forth to a variety of national currencies.

Crypto-currency — Bitcoin

Since 2009, a virtual (“crypto”) currency has become popular among hackers.  Bitcoin is a pseudonymous (that is, a fake online ID) cryptographic currency, used by the hacker underground to buy and sell everything from servers to drugs to cellphone jammers.  Bitcoin is a real currency, once valued about 3 times higher than the US dollar; its value has fallen much lower in recent years (between $4.50 and $5.50, in 2012), but has rebounded: over $950 as of 12/2016.  It has become the standard currency on Silk Road (an underground online market, mostly for drugs) and some porn and gambling sites.  (See BitCoincharts or Preev for current values.)  Bitcoin is achieving some respectability of late, and some legitimate vendors now accept Bitcoin. (Show EFF donation page.)  Bitcoin is not the only virtual currency; Litecoin and others are starting to show up (2013).

Unlike other currencies, Bitcoin (and other forms of digital or crypto currency) is not tied to any central authority.  It is designed to allow people to buy and sell without centralized control by banks or governments, and it allows for pseudonymous transactions that aren’t tied to a real identity.  In keeping with the hacker ethos, Bitcoin has no need to trust any central authority; every aspect of the currency is confirmed and secured with strong cryptography.  Individuals earn Bitcoins by selling stuff, or by mining (donating CPU cycles to organizations for profit) and store them in their Bitcoin wallet — a data file containing private crypto keys.  User can cash in their Bitcoins for traditional currency at various exchanges.

Bitcoin Central, a Bitcoin exchange that is popular in the Eurozone, claims it has secured approval from regulators to operate as a bank under French law.  (Paymium is the French company that operates Bitcoin Centeral.)  Euro-denominated funds will be insured by the Garantie des dépôts, the French equivalent to the US FDIC.  The accounts will also be integrated with the French banking system, so users can have their paychecks automatically deposited into their accounts and converted to Bitcoins.

For technical details and information on how to use bitcoins, or become a Bitcoin miner, see Bitcoin.it (especially the FAQ section) or this 22 minute Bitcoin video.

Fighting Back

In January 2006, Business Week reported on the use of the e-gold system by Shadow Crew, a 4000-strong international crime syndicate involved in massive identity theft and fraud.  One person reportedly connected to Shadow Crew as an e-gold customer and has moved amounts ranging from $40,000 to $100,000 a week from proceeds of crime through e-gold.

To combat this type of crime CMU researchers propose using a slander attack, in which an attacker eliminates the verified status of a buyer or seller through false defamation.  By eliminating the verified status of the honest individuals, an attacker establishes a lemon market where buyers are unable to distinguish the quality of the goods or services.

The researchers also propose to undercut the burgeoning black market activity by creating a deceptive sales environment, by a technique to establish fake verified-status identities that are difficult to distinguish from other-verified status sellers making it hard for buyers to identify the “honest” verified-status sellers from dishonest verified-status sellers.  "So, when the unwary buyer tries to collect the goods and services promised, the seller fails to provide the goods and services.  Such behavior is known as ripping.

I’m not sure if it counts as fighting back, but a new Trojan horse program discovered in mid-2011 seeks out and steals victims’ Bitcoin wallets, the same way other malware goes for their banking passwords or credit card numbers.  The malware, Infostealer.Coinbit, is fairly simple: it targets Windows machines and zeroes in on the standard file location for a Bitcoin wallet.  It then e-mails the wallet to the attacker by way of a server in Poland, according to Symantec.  The first actual theft of Bitcoins was reported in June 2011 by a user who claimed a hacker transferred 25,000 BTC from his machine, theoretically worth about $500,000 at 2011 exchange rates.  Another bit of malware called Stealthcoin debuted in 2010 that’s designed for turning a botnet of compromised computers into a covert parallel Bitcoin mining machine.

Zcash

Bitcoin is only pseudo-anonymous: you can create false identities, but there is a danger those might be linked back to you.  A new type of digital currency attempts to address this flaw by using modern, strong encryption as part of the protocol.  In additional to truly anonymous transactions, Zcash has a group of humans to check and support the software and protocol.

With security baked in, Zcash offers another benefit over Bitcoin: a user can remove some of the anonymity if they wish, to selected 3rd parties.  (Example: students can show their parents (but nobody else) how they spent some of the Zcash on educational expenses.)

(See these IEEE Spectrum articles about Zcash for more information.)

Lecture 2 — Security Concept Definitions (in no particular order)

Define Layered Security Architecture: multiple levels (or layers) of protection, so that if one is compromised there are others to provide protection.  An example of this principle is to use proper permissions on files, but also don’t allow root to remotely connect to your machine.)  This principle is also known as rings of security, and is related to defense in depth (see below).

Question:  Which would you rather have, one virus scanner that is 95% accurate, or three scanners from different providers, where each only offers 80% accuracy?

It turns out that three independent virus scanners, each with only 80% accuracy, is better protection than a single virus scanner offering 98% accuracy!  Of course, the same scanner run three times is not any more secure than running it once, so you must be certain that each scanner operates differently to get the protection level you desire.

A related notion is defense in depth.  This refers sharing the security burden over many parts of a system, rather than having a single system manage security:

·       An application should be written securely and deployed in a secure fashion.

·       An application/process should manage security of its data.

·       A kernel should provide security services to applications, and should enforce security for access to system resources (files, memory, network, GUI, ...).

·       Every host should have firewall and other protections, based on policies defined for that specific host.

·       Each LAN switch and router should implement security for each LAN in the organization.

·       Different collections of LANs in an organization had different security needs and policies.  Routers should enforce these.

As you can see, if defense in depth is adopted, no single firewall or other security product you can buy is ever going to be sufficient.  (No matter what the vendor promises! :-)

Defense in depth applies to the big picture too, the overall information systems’ security (IS security).  Physical security controls access into the building, keeps track of who comes and goes, detects and responds to potential intruders.  Personnel security vets individuals through suitability criteria in accordance with company or government policies, and monitors employee behavior and well-being.  (Security and HR often have a close working relationship based on the need to identify potential personnel security risks and mitigate them before they have a chance to materialize.)

Program security focuses on protecting the key elements of a company’s programs: intellectual property (“IP”), customer information, or government secrets.  Security education and training works to earn true buy-in from employees, converting them into miniature security watchdogs in their own right, dutifully monitoring themselves and their workmates.

MAC and DAC

Access controls are designed to prevent unauthorized use of resources.  Once a user is authenticated, the system consults access tables to determine if some attempted access is allowed.

Mandatory Access Controls provide a policy based security system.  This means that if the policy says some resource (file, device, port number) can only be accessed by XYZ, then not even the root user can change that!  Such policy changes must be made to policy files, which are usually human readable text files that in turn get compiled into a more efficient form.  The compiled policies are loaded into the OS at boot time and can’t be changed without a reboot.

The policy may allow users and administrators some leeway in controlling access.  Users typically are allow to grant read, write, and execute (traditional Unix permissions) to their files.  An administrator may be granted the privilege of changing the ownership of all but a few files and programs.

Discretionary Access Controls don’t have the OS enforce any central policy.  Instead, the owner of a resource (or root) is granted total control over access of their resources.  This doesn’t mean there aren’t polices, however!  PAM and other ad-hoc mechanisms exist to enforce policy.  However since the OS doesn’t make you use these mechanisms, an attacker can bypass them.

Multi-Level Security

A MLS (multi-level security) system allows an administrator to assign security levels to files, such as unclassified, secret, most-secret, top-secret, read-this-and-die, ...  A given user is also assigned a security level called their security clearance.  Then access is denied if your security level is less than that of the file (the no read-up property).  (Don’t confuse MLS with multi-layer (rings of) security.)

The Bell-LaPadula model is often used to describe MLS.  It focuses on data confidentiality and access to classified information.  In this model, the MLS system prevents a user with a high security clearance from creating files with a lower security level (the no write-down property).  For example, a process with secret label can create (write) documents with labels of secret or top-secret, and read documents with labels of secret, confidential, or unclassified.  (Note a process with a top-secret label still can’t access a document when the DAC or other MAC policy forbids it.)

Special privilege is required to declassify (assign a less-restrictive security level to) files.  Instead of declassifying documents, a higher-cleared individual can create and share sanitized documents with less-cleared individuals.

MLS is common in military and some commercial data systems; each use has, over time, defined a set of such levels.  However, sharing documents between organizations or nations is a problem, because each has different rules for labels such as “secret” or “confidential”.  MLS also requires a trusted computing base (TCB) operating system.  It is rarely enforced by operating systems today.

The Biba model is related to the Bell-LaPadula model.  However, instead of confidentiality as the goal, data integrity is.  The rules are the opposite: no write up (a less trusted process can’t create/delete/modify higher-security resources; a captain can read orders created by a major but not alter them) and no read down (a high-trust process can’t access low-trust resources, since it shouldn’t believe them anyway).

Related access control techniques designed to prevent enemy access include weighted naval codebooks and lead-lined dispatch cases that could be tossed overboard in the event of immanent capture.  Special papers could be used that instantly burn to fine ash, or are water-soluble.

Hardware Based Protection

Access controls are not the only protection mechanisms available (but they are the most useful and most used).  Hardware based protection is available as well, limiting access to resources in a hierarchical manner.  Most CPUs define such levels and assign a protection level (or domain, or ring) to blocks of memory and I/O addresses.  Intel x86 CPUs have four rings, 0 through 3.  The lower the number, the more access is allowed.  The ring level of running code is stored in a special CPU register.  The level currently in use is called the CPU mode.  In addition to memory and ports, some CPU instructions (about 15) are prohibited in higher numbered rings.

By default, applications run in the highest ring (3) and have no access to resources marked by a lower ring number.  Instead, such code must request more privileged code (the kernel) to access the resource on its behalf, allowing the system to determine if access should be allowed, to log the request, to alert the user, or some combination of these.  An attempt to access a protected resource or run a restricted instruction from a ring that doesn’t allow it, triggers a general protection fault.

While different CPUs may have many rings most OSes only use two: ring 0 (supervisor mode or kernel mode or system mode) and ring 3 (user mode).  Although rings 1 and 2 are generally unused, they are considered part of system mode if used.

While interest in supporting more than two ring levels has been traditionally low, virtualization has caused renewed interest.  By running the hypervisor in ring 0 and the various OS kernels in ring 1, the hypervisor can easily intercept resource access attempts and fool the OS into thinking it is accessing the resources directly.  Without this feature, each OS requires modified device drivers (and other changes) to invoke the hypervisor.

It may take hundreds of CPU cycles to do a context switch; that is, to change the mode.  For this reason, there is a security-performance trade-off and some OSes include application code in ring 0 to avoid the frequent context switches.  (DOS, but not modern Windows, used ring 0 for all code.  Windows runs the GUI in ring 0.  Even Unix/Linux systems are not immune to the allure of faster performance; whole web servers can be loaded as “device drivers” and run in ring 0!)

False Positives and False Negatives

These occur when checking incidents against a security policy.  A false positive occurs when the security scanner reports something as suspicious when in fact nothing is wrong or insecure.  A false negative is when the scanner fails to report a problem when in fact one exists.

Neither of these can be completely eliminated in any real scanner, and there is something of a trade-off between them.  Most scanners opt for more false positives to generate fewer false negatives.  However this can lead to the BWCW (boy who cried “Wolf!”) syndrome, leading administrators to ignore the scanners reports.  Add-on tools (usually Perl scripts) can attempt to filter log files and reports to show the most important messages, to summarize, to spot trends (dictionary or DOS attacks in progress), etc.  (Example: logwatch.)

Hostids

A Unix host is assigned a unique 32 bit ID number known as the “host ID”.  These are supposed to be unique, at least within a single administrative domain.  The number could be a serial number of the hosts motherboard or CPU, but today is often (part of) the MAC address of the first Ethernet card.  On Solaris boxes the host ID is set in the firmware (NVRAM) when the OS is installed.  (Note that re-installing may not result in an identical host ID.)

You can view the host ID with the hostid command.  Although POSIX defines a system call (kernel API) gethostid(), it doesn’t mandate any user command to return this value.  However most systems provide a hostid command.  (If not it is trivial to write a C program for this.)  Linux provides commands to get and set this value (long live open source!)

So why does this matter?  For one reason some Unix software requires the host ID registered with the vendor to validate a software license; if none (or a dup) is found software may not install (or work)!

There are far too many software vendors who love proprietary lock-in, hardware lock-in, or a combination of both (e.g., “Windows activation”).  This can be called trusted computing (in the sense of “we trust you to pay us big bucks if we give you no choice”).  A hardware-specific ID with software to query the host ID is usually how it’s done, but the host ID is set in the kernel or firmware.

When updating systems or when recreating virtual systems, you may need to “set” the host ID value.  Naturally details for doing this are not published (the vendors prefer you to buy another license) but you can find directions in various places for different distros such as in this blog post for Solaris.  (Here someone traces the obfuscated Solaris code for setting the hostid.)

[Much of the following material is from the excellent “Introduction to Computer Security” by Matt Bishop, (C)2005 Addison-Wesley]

CIA

Security requirements, whether self-imposed or mandated by an external agency or customer, are all designed to address the three fundamental objectives of computer security: confidentiality, integrity, and availability.  These three concepts are often referred to as the CIA triad.  FIPS Publication 199 defines these security objectives in precise terms, but we can define them as:

·       Confidentiality  Keeping information, resources secret from those that shouldn’t know about them.  This related to the concepts of authentication (who is requesting resources?) and authorization (what resources is a given person/program allowed to access?).  A loss of confidentiality means the unauthorized disclosure of information.  You must also consider the duration for which the information must be kept confidential; security measures that can keep secrets for a few weeks or months, may not be good enough to protect military secrets.

Correctly enforced, confidentiality prevents unauthorized disclosure of information.  The most common method of enforcing confidentiality is with encryption.  (But don’t forget other ways, such as limiting access to the data.)

·       Integrity   Keeping information and resources trustworthy (preventing improper or unauthorized modification, deletion, addition, or replacement, called corruption).  This relates to the concepts of credibility, authentication, and authorization.

It is also important to keep data externally consistent (data in system accurately reflects reality) and internally consistent (ledger books balance, totals match the sums, etc.)  A common risk with standard relational databases is losing internal consistency, which is addressed by normalization.

Integrity mechanisms can prevent corruption or detect corruption (that is, verify data hasn’t been modified).  Using message digests (hashing) is a very common way to implement integrity.

·       Availability  Keeping information and resources accessible to authorized persons, whenever they are otherwise allowed access.  This related to reliability.  One key method of availability is by using redundancy: RAID, hot-standby servers (fail-over clusters), multi-homing, and excess network capacity.  Another way to keep data available is by using backups.

There are other security concepts however, beyond CIA.  Networking router and switch vendors rarely worry about confidentiality or integrity (perhaps they should).  Auditors need to know your systems correctly implement your security requirements and policies; ensuring this is called information assurance. CIA doesn’t address concerns about authentication (who you are), authorization (who can do what), and accountability (who did what and when).

AAA

The three concepts of authentication, authorization, and accountability are often implemented in network equipment (such as routers), and is referred to as the AAA triad:

·       Authentication   Representing identity: Users, groups, roles, certificates

Authentication refers to the process of establishing the digital identity of one entity to another entity.  An entity is a client (a user, process, host, etc.) or a server (service, host, or network).  Authentication is accomplished via the presentation of an identity and its corresponding credentials.

When accessing some protected service or resource of some system you must authenticate yourself (prove who you are) before the system can decide if you should be allowed access.  There are three common ways, or factors, to prove identity.  (Additionally, you can have someone known to be trustworthy vouch for you; this is related to the notion of trust.)  To prove identity, one or more proofs are submitted; usually each is a different factor.  A requirement to present two factors to prove identity is called two-factor (or dual-factor or TFA) authentication; with more than two, it is called multi-factor authentication (MFA).  The three factors are:

·       Something you know (such as a password or credit-card number)

·       Something you possess (such as a key-card, smart-card, dongle, or key-fob)

·       Something you are (biometrics, such as a fingerprint)

The most common method used is by providing information only you and the system know (i.e. a password, a.k.a. a shared secret).  Note the system doesn’t have to know the secret; it is enough if they can verify you know the secret.  (This is why plain-text passwords don’t have to be stored on a server.)

Using a phone-based system (i.e. providing a phone number that can be checked) can be used to call-back the user trying to access a resource to confirm it really is them.  Phone calls are made to the (presumably real) user when someone attempts to log in as him or her.  The user can then punch in a code on the phone.  Phones an also serve as fraud alerts.  Email call-back (“verification email”) systems are also commonly used, especially when resetting passwords.

Another possibility is proving possession of something only you (should) have, such as a smart card swipe card, credit-card (using CVS number), dongle, or RFID.  Using a key fob that displays a number that changes every minute, that only the server and the fob know, is another example (RSA SecurID, YubiKey).

Cell phone apps can take the place of dongles or fobs.  Note that since modern cell phones (2012) only run one app at a time, you can’t use the AWS-MFA app (“virtual dongle”) and the AWS console app!

Gmail has an option for two factor authentication, but it had (1/2013) a back-door built-in that could be used to not only bypass the extra step, but could be used to bypass authentication altogether!  Called application specific passwords, using these other passwords would disable TFA, useful when fetching mail from, say, Thunderbird.  But since the password was sent in plain text, it could be intercepted and used.

Increasingly biometric identifiers such as a fingerprint, eye print, handprint, voiceprint, or even lip prints may be used.  (Imagine having to kiss your computer to log in.)  Increasingly, ATM machines use vein prints of your hand.  Biometrics have some problems; they are not yes or no.  Like spam scanners, they have problems with false positives and false negatives.  Depending on the situation, you can tune the scanner to favor one or the other.  Another problem is when the biometric data on file for you has been stolen.  The bank can’t just issue you a new fingerprint!  Another problem is fake biometrics: fake fingers (or in one gruesome case, real fingers cut from a victim) or special latex gloves can defeat advanced fingerprint readers, including ones with “liveness” tests such as sweat, temperature, or pulse.

The next generation of biometrics include continuous monitoring, making sure a living person is still there: pulse, temperature, skin conductivity, heartbeat monitors, even ECGs (brainwave monitoring).

As mentioned above, another technique is to rely on (“trust”) another organization for authentication.  You present proof (a credential) that the organization has authenticated you, by showing a valid passport, driver’s license, major credit-card, Kerberos ticket, etc.  Proving you have a valid email address or phone number falls into this category.

Many of the most convenient authentication schemes are weak and easily circumvented (spoofed).  When using such weak methods, users typically need to use a two-factor (or multi-factor) authentication system.  For example, entering something that only they would know, and use something that only they would physically have on their person.  This means entering a login and PIN or password, along with the use of a USB authentication key or smart card, for example.

CAPTCHAs — Proving you are Human

Some systems don’t care who you are, as long as you are a person and not some “bot” or program.  Services such as web-based guest-books or the WHOIS database are available to anyone, but not to automated programs (robots or bots).  In these systems, you must only prove you are human.  Typically this is done by having the use do a task easy for most people but difficult for machines.

A CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart, if you can believe that) is an obscured graphic of some text, shown on a web form.  A human can usually read and then enter the text, but a program has a difficult time reading the text.  (Captchas have become an incentive to improve OCR systems!)  Other tasks include picture recognition (e.g., “click on the one picture above of a moose”), math problems stated in English (show Ars Technica test, original URL: http://contact.arstechnica.com/spammy/40706/), or audio CAPTCHAs.  SANS Institute (Chief Research Officer Johannes Ullrich, Sys Admin, V16 N4 (April 2007), “Minimizing Content Form and Forum Spam” pp.30–ff) reports that CAPTCHAs are effective at reducing spam, but also reduce legitimate traffic by about 20%.

NetworkWorld.com 11/1/11 - Researchers from Stanford University have developed an automated tool that is capable of deciphering text-based anti-spam tests used by many popular websites with a significant degree of accuracy.  With their tool, the tests used by Visa’s Authorize.net payment gateway could be beaten 66 percent of the time, while attacks on Blizzard’s World of Warcraft portal had a success rate of 70 percent.  (The only tested sites where CAPTCHAs couldn’t be broken were Google and reCAPTCHA.)  Authorize.net and Digg have switched to reCAPTCHA since these tests were performed.  These researchers have also successfully broken audio CAPTCHAs on sites like Microsoft, eBay, Yahoo and Digg.

The Stanford researchers came up with several recommendations to improve CAPTCHA security.  These include randomizing the length of the text string, randomizing the character size, applying a wave-like effect to the output, and using collapsing or lines in the background.  Another noteworthy conclusion was that using complex character sets has no security benefits and is bad for usability.

·       Authorization  Policy stating who is allowed to do which actions on what resources: ACLs, capabilities (tickets).  Authorization refers to the granting of specific privileges (including none) to an entity (e.g., process, host, AS, or user).  This is based on their authentication (proven identity), what privileges they are requesting, the current system state (e.g. the service is available), or on restrictions such as time-of-day restrictions, physical location restrictions, or restrictions against multiple logins by the same user.

·       Accountability  Keeping track of who does what and when.  Log files are an example, as is the tracking of the consumption of network resources by users.  The data generated is used for management, planning, billing, auditing, or security (e.g., log monitored by an IDS).  Typical information that is gathered includes the identity of the user, the nature of the service delivered, when the service began, and when it ended.

Information Assurance (IA)

Over the years, it has become apparent that CIA alone (or AAA alone) is not sufficient.  The result of this has been a new acronym: IA.  IA is the set of controls and processes, both technical and policy, intended to protect and defend information and information systems by ensuring their confidentiality, integrity, and availability, and by providing for authentication and non-repudiation (see below).  IS (Information Security) is also known as Information Assurance and Security (IAS); in U.S. government/military circles, the new hot term for INFOSEC and SIGINT is “Cybersecurity”, but that is just another term meaning “Information Assurance” (IA).  Some U.S. Federal jobs now require IA certification, and IAS is now (2013) a core knowledge area of a four-year computer science degree program.

From an IA perspective, there are five pillars (or core concepts): confidentiality, integrity, availability, authentication, and non-repudiation.  All but that last were discussed above.

Non-repudiation is related to accounting (and data integrity).  This means you can’t convincingly deny some action such as sending a message, receiving a message, entering/changing/deleting data, etc.  (Example: ordering an expensive item on-line, then refusing to pay claiming you never ordered the item.)

Other important concepts include:

·       Security Policies  A security policy is a statement of what is and what is not allowed.  This is also called a specification.  Policies can be stated by mathematics and tables that partition the system states into allowable (secure) and not allowed (insecure) states.  More commonly, vague descriptions are used, which can lead to ambiguity (some states are neither allowed nor disallowed, or both).

When two or more entities communicate or cooperate, the combined entity has a security policy based on the individual policies.  If these policies contain inconsistencies the parties involved must resolve them.  (Example: proprietary document provided to a university.)

·       Polyinstantiation is the concept of creating a user or process specific view of a shared resource.  (So Process A cannot affect process B by writing malicious code to a shared resource, such as /tmp.)  The term refers to a similar concept for databases, where different users get different views of a database (think of a virtual table).  (For cryptography, polyinstantiation means to have a copy of a secure key in more than one location.)

The pam_namespace module creates a separate namespace for users on your system when they login.  This separation is enforced by the Linux operating system so that users are protected from several types of security attacks.  Using this module, PAM creates a polyinstantiated private /tmp directory at login time.  This is transparent to the user logging in; the user sees a standard /tmp directory and can read and write to it normally.  However that user cannot see any other user’s (including root’s) /tmp space or the actual /tmp file system.

The kernel supports polyinstantiation too.  /proc/self gives a private view of kernel resources.  SELinux also supports this, with special mount options.

Security Mechanisms

A security mechanism (or security measure or security primitive) is any method, tool, or procedure used to enforce a security policy, or to reduce the impact (and frequency) of threats.  Not all mechanisms are tangible.

Security mechanisms can be used to prevent an attack, to detect an attack has occurred (or is occurring), or to recover from an attack.  Often you will use multiple mechanisms working together to meet all three of these goals.

Qu: what is the goal of a password mechanism?  (Ans: prevention, audit; there’s overlap.)

Prevention mechanisms keep the system functioning normally and available during any attack.  Such prevention mechanisms include over-provisioning and fault-tolerance, and can be used in safety-critical systems where the high cost is deemed acceptable.

Detection mechanisms are used (in addition to the obvious) to monitor the effectiveness of other mechanisms.

Recovery mechanisms are deployed when detection of an attack has occurred.  The first part of recovery is to repair the damage done by the attack and to restore normal service levels.  (Prior to that, you need to stop an on-going attact.)  Recovery is not complete until an assessment of the incident has been made and preventative measures are put into place.  This may include a change to policy, design, or configuration of services, the addition of extra mechanisms, or legal action.  (There will likely be PR actions needed as well.)

·       Dual Controls These are mechanisms designed to prevent any single individual from violating a security policy.  Examples include dual-entry bookkeeping, safe deposit boxes at banks, and military missile-launch protocols (require two persons to turn keys more than 10 feet apart at the same time).

·       The meaning of trust and Assurance      Trust is a measure of trustworthiness of a system, relying on sufficient credible evidence that a system will meet a set of given requirements.  For example, a system may be considered physically secure if the system is locked up and only authorized people have keys.  However, this assumes that those with access to the system and who know how to pick locks (or copy keys), can be trusted not to do so unless authorized.  Bank managers are authorize to move funds between accounts (up to some limit), but must be trusted not to move funds into their private accounts.  You pay for some goods on amazon.com, but must trust them to actually ship them to you.

Another example is that you must trust some installed server or other software not to allow violations of system security policy.  If the server has exploits (e.g., bugs, back-doors, viruses) that cause the server to fail and break our security, our trust was misplaced.

Trust relationships exists whenever there is some risk and uncertainty.  One must weigh the risks of granting trust against the expected gains.  Another way to define trust is “a positive expectation regarding the behavior of somebody or something in a situation that entails risk to the trusting party”.

Assurance is the measure of confidence that a system meets its security requirements, based upon specific evidence gathered through various assurance techniques.  While trust can’t be precisely measured or even defined, circumstantial and anecdotal evidence (assurances) can be accumulated that can be used to determine how much to trust a system (or determine your insurance rates!)

Measures that can be taken to provide assurance in a trusted system include security cameras in high security areas to make sure no one is using lock-picks, extensive background checks before hiring a bank manager, following standard best practices, obtaining certifications, using vulnerability scanners, using referrals and recommendations for software, and tamper resistance.  For example, don’t trust a bank manager if they have stolen before, or buy goods on-line from a vendor with a poor reputation, or install software that has a history of problems (e.g., MS IIS) or is from an unknown source (e.g., acme email server).

Consider installing a strict (SELinux or other) MAC system.  Now you don’t need to trust software as much, because the (trusted) MAC system provides assurances that broken or malicious software won’t compromise the security policy.

·       Design    The design of a system is the process of determining a set of components (mechanisms including procedures) and their interactions and configurations, that will be used to implement the security policy (specification).  The design that results can be analyzed to determine if it will satisfy the specification.  While it may be difficult or impossible to fully answer this question, assurance can be generated that the design will satisfy the security policy.

Once this is done, the design can be implemented.  The implementation must satisfy the design, much like the design must satisfy the specification.  Proving an implementation satisfies some design is difficult and involves proofs of correctness.

·       Identity  is the representation of some unique entity or principal: a person, a website, an authorized meter-reader).  Note a given entity may have many identities.  (How many login names do you have?)

·       Privacy  is a complex notion, and not always a person’s right — it depends on local laws and customs.  Each person or entity (corporation or government agency) has their own unique point of view on what information is (or should be) under that person’s or entity’s control.

Medical records are a case in point: Doctors don’t like to release records to a patient, in case there is later any dispute.  By U.S. law, some records must be released if a patient wants them.  In addition, medical records can be used for research purposes and medical assurance purposes.  In these cases, identifying information is supposed to be removed first, in a process known as anonymizing or sanitizing or blinding the data.  But this is extremely difficult to do right, and often impossible when only a few records are involved.  (The British Medical Association has the most comprehensive laws and rules on patient privacy.)

Identity theft is merely one problem.  Credit reports are not only a pain over which most of us have little control, but the failure of the reporting companies may in fact cost people employment.  An abusive ex-spouse, a company reviewing the movies you watch to determine your employment prospects, etc., are all legitimate reasons to desire privacy.

While privacy laws exist to remove PII from publicly available information (including HIPAA and FERPA), doing so is very difficult.  [Much of this section was adapted from a post made on Ars Technica by Nate Anderson, 9/8/2009.]

The Massachusetts Group Insurance Commission had a bright idea back in the mid-1990s—it decided to release “anonymized” data on state employees that showed every single hospital visit.  The goal was to help researchers, and the state spent time removing all obvious identifiers such as name, address, and Social Security number.  At the time GIC released the data, William Weld, then Governor of Massachusetts, assured the public that GIC had protected patient privacy by deleting identifiers.  In response, then-graduate student Sweeney started hunting for the Governor’s hospital records in the GIC data.  She knew that Governor Weld resided in Cambridge, Massachusetts, a city of 54,000 residents and seven ZIP codes.  For twenty dollars, she purchased the complete voter rolls (often used to link data from anonymized data sets) from the city of Cambridge, a database containing, among other things, the name, address, ZIP code, birth date, and sex of every voter.  By combining this data with the GIC records Sweeney found Governor Weld with ease.  Only six people in Cambridge shared his birth date, only three of them men, and of them, only he lived in his ZIP code.  Dr. Sweeney sent the Governor’s health records (which included diagnoses and prescriptions) to his office.

When AOL researchers released a massive dataset of search queries (2008?), they first “anonymized” the data by “scrubbing” out user IDs and IP addresses.  When Netflix made a huge database of movie recommendations available for study it spent time doing the same thing.  Despite scrubbing the obviously identifiable information from the data, computer scientists were able to identify individual users in both datasets.

In AOL’s case, the problem was that user IDs were scrubbed but were replaced with a number that uniquely identified each user.  But those complete lists of search queries were so thorough that individuals could be tracked down simply based on what they had searched for.

The Netflix case illustrates another principle, which is that the data itself might seem anonymous, but when paired with other existing data, reidentification becomes possible.  A pair of computer scientists famously proved this point by combing movie recommendations posted on the Internet Movie Database with the Netflix data, and they learned that people could quite easily be picked from the Netflix data.

Just because of high profile commercial failures does not mean that anonymization is impossible.  This is a heavily researched area in computer science and statistics right now (particularly in the database community; search for terms like “K-anonymity” and “L-diversity”).  The census bureau generates publically available anonymous data, with no reported breaches of this data.  However the data is of limited use because of the intensive data scrubbing.

There are some steps you can take to help keep data anonymous:

·       Never use any part of the DOB.  Use the person’s age, rounded to a whole number of years, and modify that by a random factor of +/- two years.  The resulting data should still be statistically useful for most purposes.

·       Never use zip code.  Use the person’s county or province.  Better approach: use a quad system by breaking the state(s) into 4 or more arbitrary areas.  Best approach: don’t use a system which targets user locations.

·       Never store a user’s account numbers, even blinded ones.

·       Read the reports on privacy that are available, such as those from the BMA, and use the advice.

Case Study (adapted from How to De-Identify Your Data by Angiuli et al, published in CACM 12/2015):

MOOC courses like to release their student data to help researchers improve the courses.  To allow this under FERPA, the lawyers for edX decided to use k-anonymity with k=5 (based on advice from the U.S., DoE).  This means every record must have the same identity-revealing traits (quasi-identifiers) as at least four other students in the data set.  The first step is to identify which fields in the records are quasi-identifiers.  They decided on: course ID, level of education, year of birth, gender, country, and number of forum posts made.  (Course ID because a given student might have taken a unique combination of courses, and number of posts since the forums are public and by simple counting, re-identification might be possible.)

There are two ways to achieve this.  One is by generalizing quasi-identifiers into groups, for example, instead of the raw data years old, you group students into “buckets” such as “18-21, 22-25, and 26-29”.  If a three year bucket doesn’t result in sufficient anonymity, you can increase the bucket range.

The second method to anonymizing the data is to delete any records that aren’t sufficiently anonymous after generalization is done for all quasi-identifiers.

As you can see, de-anonymizing data properly is a lot of work to do well.

·       Risk analysis   (Sometimes referred to as Cost-Benefit analysis or CBA, or trade-off analysis or TOA.)  This means to determine whether some asset should be protected, and if so to what degree.  You must balance the cost of an incident should it occur, the likelihood that it will occur, and the cost of preventive measures.  You must also consider mitigation measures, which are mechanisms and policies designed to lower the cost and/or likelihood of some type of security incidents.

Risks change with time, and thus a new risk assessment should be made periodically.

·       Audit trails and Logging    Audit trails provide Accountability, which prevents false repudiation (“I didn’t do it!”).

·       Intrusion Detection   Host IDSs (e.g. file alteration/integrity monitor, or FAM/FIM) and network IDSs seek to detect when an intruder has been attempting (or successful) at compromising integrity.

IPA — Identity, Policy, Audit

For efficiency, compliance, and risk mitigation, organizations need to centrally manage and correlate vital security information including:

·       Identity (hosts, virtual machines, users, groups, authentication credentials)

·       Policy (configuration settings, access control information)

·       Audit (events, logs, analysis thereof)

Mid- and large-scale organizations today need to set up centralized identity management.  IPA is also known as Identity Management (IdM).

Policy means a broad set of things, including access control policy, MAC policy, security configuration settings, which packages and patches are applied and running, and what system configuration settings are set.  Organizations want to be able to centrally manage this information applying different policies based on machine group, location, user, and more.  Finally, organizations need to be able to gather and analyze log and audit data and they need to meaningfully parse that data without getting overwhelmed.

The problem is all three of these aspects of security are inter-related.  There are too many “ad-hoc” implementations of each aspect to manage and administer easily.  Often custom shell scripts are needed to tie all the different systems together.

FreeIPA (FreeIPA.org) is an integrated security information management solution combining Linux (Fedora), LDAP (Fedora Directory Server), Kerberos, NTP, and DNS.  It consists of a web interface and command-line administration tools.  This is essentially what Windows Active Directory is: a collection (“domain”) of users and computers, trust relationships to other domains, and authentication and authorization rules for users.  FreeIPA provides identity managment features, including single sign-on, for modern Red Hat-like systems.

Lecture 3 — Security Issues of Backups, Installs, Updates, and Patches

Treat backup media as very sensitive security.  Ideally you need to destroy completely old media, never just throw away (dumpster-diving).  (See this YouTube video of a disk shredder in action.)

Never be the first to install new stuff.  (Policy should state how long to wait before installing new packages.)

Always get packages from trusted sources.  If you really feel a need to install new software not available from a trusted source, be very sure you really need that software!  Do not follow any links you find in public forums or in spam.  Instead lookup the software yourself with Google and go to the author’s website.  Check the DNS info for that site and make sure it wasn’t added yesterday!

Verify the download no matter where it came from (using gpg or if not available at least md5sum).  If no verification is available, then try to download another copy of the package from a completely different location (preferably from a different country) and compare the packages with cmp.

Installing from source code can be safer than installing a binary package, if you have the skills and time to carefully examine the code (including any auxiliary code such as makefiles and configure scripts).  If you don’t have required programming skills but the software is key to your organization, consider hiring a consultant.

Never compile code as root!  After building, try testing the code as a non-root user before becoming root and running make install.  Make use of the “-n” option to make to see what the make install command tries to do.

When running new/untrusted software, use tools such as strace and ltrace (which traces both sys-calls and library calls) and, additionally on Solaris and BSD, dtrace, truss and sotruss:

  ltrace -fS -n 2 -a 70 -- date 2>&1 >/dev/null |less

Look for odd system calls (e.g., open a network socket when running the date program would be suspicious, as are exec calls to /bin/login, ...), odd libraries (.so files in non-standard locations, or non-standard .so files anywhere), and odd arguments to system calls (very long strings, strange pathnames, etc.)  Note these trace commands will show in plain text everything passed as arguments, including usernames and passwords!  Tracing tools are invaluable learning and trouble-shooting tools; they are also great security cracking and investigation tools.

To contain untrusted code you should use virtual hosts VServer, VMware, ... if available.  If not then in addition to using the security features you do have (chroot, jail, zones) you can use sandbox software, a.k.a. system-call censor software, that intercepts each system call, inspects it, and then allows, denies, or asks the user (useful for trusted but vulnerable software).  You can define a policy file for each program to run in the sandbox, defining the minimum access required.  Examples include Janus (user-level censor) and Systrace (kernel-based censor).

Updates generally replace one version of a package with a newer one.  All files are replaced with an update, so you must treat an update the same way as new software, that is untrusted.

Patches come in two types: binary and source.  A source patch modifies the code and then requires a re-build, so the same concepts apply as above.  Binary patches edit the machine code files directly.  Never use binary patches except from your system vendor!  All patches (of either type) are installed according to your patch management schedule.

Boot Security

On *nix systems the boot process is not secure by default.  If an attacker can boot to single-user mode on most systems they get a root shell with no password required.  By inserting a bootable disk (floppy, CD-ROM, DVD, flash) the attacker can bypass all your security efforts.  There are a few steps to take to make the system boot procedure more secure:

A grub password is important.  This will ensure that the user can’t alter the kernel parameters (say to boot into single user mode).

Set the system to boot from the HD first.  This prevents a bypass by inserting a bootable media.

Password protect the BIOS.  This can still be cleared by opening the case and setting a jumper, or in some cases by removing a battery.  But this is not easy, especially if you...

...Use a secure (locking) case, and /or keep servers in physically secure and monitored locations.

UEFI Secure boot

UEFI Secure boot is a feature that Microsoft got hardware vendors to support.  It requires any boot loader or kernel software loaded by EFI to be digitally signed; the hardware (TPM) will refuse to load any code not properly signed.

The idea is to prevent malware from loading during the boot cycle, which is more complex (and harder to secure) than older BIOS boot cycles.  More likely, it will lock most users into Windows8 with no choices at all.  (X86 hardware has that “feature” enabled by default, but there is supposed to be a way to turn it off for that platform.  There is no way to disable it on other types of CPU, i.e. ARM.)

Rather than challenge this monopoly, Fedora 18 will pay Microsoft $99 to digitally sign Fedora, so that it can be installed on such hardware.

Secure Boot requires that the hardware check a digital signature of the firmware used, and have that firmware check the digital signature of the kernel it loads.  To support Linux, Ubuntu, SuSE, and Red Hat have developed a “shim” boot loader, signed by Microsoft, which in turn loads grub.

This shim bootloader, called Shim, contains a public key under Fedora’s control (or whichever vendor you use) and is signed by Microsoft.  The shim will only boot binaries signed with the public key and allows the developers to build and sign all other binaries themselves without going back to Microsoft to get bootloaders or other components signed.  (The Linux foundation has also developed a pre-loader loader that is similar to the shim loader, only it doesn’t check the signature of the full loader (grub).)

A locked-down boot environment and signed kernel do block modified bootloaders and booting attack code.  However, this won’t prevent an attacker from using a booted kernel to launch another kernel (or malicious LKMs).  To ensure that doesn’t happen, direct hardware access from userspace will be blocked.  Utilities wanting to access hardware must go through kernel modules that have been signed by a key that the kernel trusts.

SUSE has assisted the process by developing useful elements such as a simple mechanism to allow new keys to be enrolled from within a booted operating system, avoiding reliance on what is expected to be the variety of inconsistent user interfaces from PC firmware makers.

Shim will also notice if the second stage bootloader is signed with a key it doesn’t recognize.  If so, users will have the option of enrolling a new key from install media and having it added to a trusted key list.  Sysadmins can also deactivate signature validation in the shim, so developers need not sign kernels they are developing.  This requires a reboot and a user generated password to be activated.

Security of LOM (Lights-out Managment)

Intel developed IPMI, a popular open standard for implementing LOM, supported by Dell, HP, and others.  Intel replaced IPMI with Active Management Technology (AMT), and later with Intel’s Management Engine BIOS Extension (MEBx).  You can usually get to that by powering up while holding down the F12 key.

MEBx actually uses a separate CPU (a system on a chip computer) that runs digitally-signed code from Intel, meaning end users have no control over the system and no way to audit the code for vulnerabilities.  Several vulnerabilities were reported in 2018.

LOM can run even when the main system is turned off or crashed and can invisibly use your NIC if there isn’t a separate LOM-dedicated one.  The only requirement is access to ports 16992 and 16993.

It is strongly suggested you change the default password for MEBx on your PC if it has that feature.

Lecture 4 — Security Threats

A threat is a potential violation of security.  (U.S. Military defines a threat as a potential intruder.)  This can also be called a vulnerability.  Note that the violation need not occur, but just be possible, to be considered a threat.  You must guard against threats.  The common counters to threats are the three security services (CIA) defined above.

An action that could cause a threat to occur is called an attack.  Those who perform such actions, or cause them to be performed, are called attackers or sometimes intruders or crackers.

Threats come in all shapes and guises, but can be classified in a number of ways.  One such way divides threats into four categories:

·       Disclosure (unauthorized access to information).  Snooping/sniffing (passive wiretapping), stealing passwords or data, are examples.

·       Deception (acceptance of false data).  Social engineering is an example.  Spreading false information deliberately, called disinformation, is another (e.g. the Internet Water Army).  Companies may use disinformation to hide their true timetables for product launches, disrupt competitor’s sales (e.g., leaking rumors about upcoming products or bad reports on the competitor’s product), or protect information.  (E.g., in WWII, to protect the fact that they had broken the Nazi encryptions, false but plausible stories would be leaked about how bombers “just happen” to stumble upon Nazi convoys.)

·       Disruption (prevention of correct operation of a system).  DOS, web site defacement.

·       Usurpation (unauthorized control of some part of a system).  Having your mail system used for spam by a virus.

Additional classifications of threats

Some threats fall into more than one of these categories.  For example, modification or an unauthorized change of information is a threat where the goal may be deception, disruption, or usurpation.  (This is sometime referred to as active wiretapping.)

An example of this type of threat is the man-in-the-middle (“MitM”) attack, where Anne thinks she is communicating with Betty, but Cindy is intercepting Anne’s messages and forwarding (possibly altered) messages to Betty.  (And in the reverse direction too.)

Masquerading or spoofing is when one entity (person, website, ...) impersonates another.  This type of attack falls into the deception and usurpation categories.

Delegation is when one person (or server) authorizes another to act on their behalf.  Note the distinction between delegation and masquerading: in the first, no one pretends to be someone they are not, and if Al delegates to Bill, Bill will not pretend to be Al when communicating with Charlie.  Bill will inform Charlie he is Bill, acting for Al, and if Charlie queries Al, he will confirm.

Repudiation of origin is a false denial that an entity (person or server) created or sent something.  An example is when you order some product, it is shipped to you, and you refuse to pay, claiming you never ordered the product.  This is related to

Denial of receipt is a false denial that you received information, message, or some product or service.

Delay and Denial of Service are threats that inhibits the legitimate use of some service or resource.  The difference is in the time of the outage; a delay is for a short while, or applies to a single message.  This form of attack usually is used when one has compromised a secondary server.  By delaying a response from the primary server (whose security the attacker may not be able to compromise), many clients will retry using a secondary server (which the attacker may already be in control of).

A buffer overflow attack is an attempt to exploit poorly written software, to make it do things it wasn’t intended to.  An example is a server program, run as root.  If you can pass enough bad data to it, you may be able to trick the program to, say, start a root telnet session for you.  (See p. 308 for details.)

Rootkits: A typical system has so many services running, written by so many different people, and each configured separately, that it is perhaps inevitable that vulnerabilities will exists on your systems.  Many systems have well-known sets of these, and attackers have created toolkits designed to crack a system and grant the attacker root privileges.  These have evolved over time into point and click programs to crack systems, and are called rootkits.  These are popular with script kiddies, novice attackers who don’t know about security except how to run some rootkit.  (Show HackAttack.txt.)  There are frameworks available for hackers to use when developing rootkits, and to perform some pen testing, including Metasploit, Zeus, and Blackhole.

Rootkits provide several “features” including file and directory hiding, process hiding, socket (open port) hiding, back-doors, and log file filtering.  These are done by modifying the kernel (say by loading some infected module) and replacing many standard system utilities with hacked versions.

Sometimes the modified code is hidden to make it hard to remove.  It can hide in any NVRAM on the system (most PCI devices have some, not to mention BIOS) or in unused disk sectors.  (There are lots of hidden sectors on IDE disks, and some on any disk/partition/filesystem.)

The only 100% failsafe method of recovering from a system level compromise is to completely reinstall the box, and carefully examine any data that is restored from a backup and all NVRAM.

There are several rootkit detection methods but none are 100% reliable.  Building a custom kernel increases security; attackers creating rootkits (and worms) often rely on known kernel (version-specific) RAM addresses.  With a customized kernel, the addresses will be different.  (Most likely result would be a panic.)

Reverse engineering is a form of attack where the attacker analyses the executable, looking for ways to either exploit it for system attacks, to steal the intellectual property of the author, or to bypass restrictions on the use of the software.  Naturally, this is easier if the source code is available.

Network Attack Types

Here is a brief list of the types of attacks on communication networks [Stallings Cryptography and Network Security 4th ed., p. 319]:

Disclosure             When an entity (person or process) not possessing the appropriate key can access the message contents.

Traffic analysis    Learning the pattern of communications between parties not including yourself.  When, how often, the duration, and the size of data passed during communication can be valuable data to an opponent.  Another possibility is message injection (“AF is Midway” WWII story).

Masquerade         Messages with a fake source, including fake acknowledgements (or negative acknowledgements, the ACK of non-receipt).

Modification         Changes to the content, including additions, changes, and deletions.

Re-sequencing      Most messages today are sent as a sequence of packets.  Sequence modification, or re-sequencing, means the insertion, removal, or reordering of these packets.

Time modification  The delay or replay of messages or parts of a message.

Repudiation          Source repudiation is the denial of transmission of a message by the alleged source.  Destination repudiation is the denial of receipt of a message of the alleged recipient.

Network malware has two parts, the payload and a propagation mechanism by which it spreads.  While the payload can be anything, there are only a few ways to propagate.  So another way to classify network threats is by how they propagate:

·       A virus is computer code that attaches itself to applications (any other executables and DLLs).  When the executable code is run, so does the virus code, which then tries to infect other applications in addition to whatever evil for which it was designed.

·       A worm is stand-alone malware; it doesn’t need any host application.  Often they are self-propogating, not requiring any user action to spread to other hosts.  Worms are usually RAM-based processes, which run on one host and attempt to invade other hosts via the network.  A worm usually arrives on your host in the payload of a virus.  (Se RFC-1135 for a discussion of the famous Morris worm.)

·       A trojan horse (or just trojan) is similar to a virus, except it doesn’t infect other random software.  Instead, it is attached by the creator to some application that a user would want to download and install.  When it is run by the user, in addition to some useful task, it runs the payload too.  If attractive enough, users will download (or even purchase) the trojan, and run it often; once is enough to deliver the payload however.  (Sony put out music CDs that, if played in a computer, would install a rootkit.)

In practice, modern malware uses multiple propagation methods, sometimes not even needing to infect executables to do the damage.  (Malicious data can trick trusted code into doing bad things; think SQL injection attacks for example.)

Common Vulnerability Scoring System (CVSS)

Another way to classify threats is the Common Vulnerability Scoring System.  CVSS is used to classify threats on a 10-point scale of severity/urgency.  It is widely used.  Read more about it at its home, www.first.org/cvss.  The CVSS numbers included in various databases of threats and vulnerabilities don’t include some temporal or other changing parameters, so the scores shown may not be completely accurate.  The NVD includes a CVSS calculator you can use to determine an accurate score for threats of interest to you.

Other Vulnerabilities

There are many different types of vulnerabilities in any complex system.  A few that are often overlooked include:

Too much data can overwhelm a human’s or program’s ability to process it.  For example, a current assessment indicates there were warnings available prior to the 9/11/2001 attack but those were lost in a “flood” of data.

Math errors are vulnerabilities in the hardware and common software used for INFOSEC.  Such errors have been found in older systems, such as the division bug in Intel’s Pentium microprocessor discovered in 1994 and a multiplication bug found in Microsoft’s Excel in 2007.  A math error would allow an attacker to break a public key cryptography system by discovering the error in a widely used chip and sending a “poisoned” encrypted message to the computer, allowing the attacker to compute the value of the secret key used by the targeted system.  (Show the Mitre top 25 software errors, and the SAMATE reference DB, from class resources.)

User programs access kernel APIs in a CPU dependent way.  In essence a table of kernel system call addresses is built, and a program wanting to make a kernel call will put the index number of that call in a predefined CPU register, then executes an instruction (e.g. TRAP, that will cause an interrupt on older systems, or SYSENTER on more modern hardware), which will switch the processor from user to kernel mode.  The kernel then executes the requested system call.  The security problem lies in the passing of arguments to the kernel system call.  The name of a file for example, might get changed after being validated but before the system call executes.  *Nix kernels prevent this by making a private copy of any arguments, then validating the copy.

Lecture 5 — Cryptography Overview, Digests / Hashes, MACs, Trust, Steganography

An access control mechanism is used to support the goal of confidentiality.  Another possible mechanism is cryptography.  Some of the many forms of cryptography include: one-time pad (OTP), symmetrical (crypt, DES), and public key (RSA, ECC).  These methods encrypt plaintext using a standardized algorithm, plus a password, passphrase, or key.  The result is known as ciphertext; even knowing the algorithm details but without the password/key, it is computationally infeasible to compute the plaintext.  However if you know the key, it is relatively simple to recover the plaintext.  So no access control is needed to protect the ciphertext from attackers.  (Unless you want multi-layered security.)

The different methods can be classified as either stream ciphers or block ciphers, or as shared/symmetric key or public key systems. These are discussed next.

Converting plaintext to ciphertext is known as encrypting.  Converting ciphertext to plaintext is known as decrypting.  (Technically, there is a difference between this and enciphering/deciphering: encryption results from using codes (also called encoding/decoding), while encipherment from using ciphers.  In practice, the terms are often used interchangeably.)

Opportunistic encryption (OE) refers to any system that, when connecting to another system, attempts to encrypt the communications channel otherwise falling back to unencrypted communications.  Many email systems support this but often other network systems on a host won’t, unless enabled.  Examples include VoIP, Kerberos, HTTP, DNS, and others.  (Windows 7 includes OE support for most network services, but it is off by default.)  OE provides many possibilities for abuse; it has been removed from most Mozilla software.

Passwords and Keys

A passphrase is just a password that accepts spaces.  (Usually, a passphrase is composed of randomly selected but otherwise valid words, and sometimes symbols are used to separate words, not spaces.)  A key is generally a huge random number, with hundreds or thousands of digits.  Keys are too hard to type in or remember and are usually stored in files, which in turn are encrypted with a password or passphrase.  To use the key, you need to enter the password/passphrase.

Random Numbers

The heart of cryptography is the use of random numbers.  (See RFC-4086, Randomness Requirements for Security.)  Basically the numbers should be evenly distributed in their range (for a uniform generator; bell-curve and other types are possible), all values in the range should be possible, and the next value generated shoujld be completely unpredictable from the list of previous values.  But, how can a computer generate high quality (truly random, or at least totally unpredictable by an attacker)?  Since computers are deterministic (unless they include special hardware), they cannot “compute” random numbers.  The numbers they produces form a pseudo random sequence, and the algorithms are known as pseudo-random number generator (PRNG).  (For example, if the PRNG uses the forumula previous value + 5 mod 101, by observing a few of the generated values it is easy to determine what the next values should be.)

FIPS-140-2 contains tests of random number sequence quality; see testrng(1).  However, such tests are for statistical properties of the stream of numbers produced (looking for patterns, or for values that occur more often than they should).  For use in security systems, the numbers must also be unpredictable.  These are produced using a truely random seed value with a cryptographic PRNG (CPRNG), to start a sequence of numbers that are unpredictable as well as statistically random.

Since 2013, the U.S. government has been operating the NIST random number beacon, to set a standard for randomness.  (But thses cannot be used for crypto, as each value is preserved and timestamped forever.  So it’s not a secret.)

In *nix, /dev/random is usually a special file that serves as a random number generator (RNG) or as a pseudo-random number generator (PRNG).  It allows access to environmental noise collected from device drivers and other sources (for example, the number of milliseconds between key presses or mouse clicks, the time between TCP packet arrivals, the seek time of the disk since the last head movement, etc.).  All these random numbers are stored in a list called the entropy pool by the kernel.  As each value is added, the kernel adds to the estimate of the available entropy.  When /dev/random is read, the data from the pool is hashed using a cryptographically secure hashing algorithm (e.g., SHA-256), the results are returned, and the estimate of available entropy is decremented.

Virtual machines have a security issue with this procedure; they only have fake hardware, resulting in low or no entropy.  If the host OS maintains an entropy pool used by the virtual machines, one VM can use up the pool, “starving” other VMs.

Modern Intel CPUs include special RNG hardware that can be used in place of the traditional sources.  The Linux VM software (KVM and libvirt) support (7/2013) a paravirtual random number generator device.  This can be used to prevent entropy starvation in virtual machines.  (See VirtIORNG for some details.)  However, due to distrust of Intel, Linux uses such hardware only to add to the entropy pool.

Although not all systems implement /dev/random this way, this method provides cryptographically secure random numbers.  However, any attempt to read more random bits than the entropy pool currently contains will block, waiting for more random noise to be added to the pool.  Linux was the first operating system to implement a true RNG this way (Theodore Ts’o’s Implementation for /dev/random in 1994).  This feature can be exploited in a side-channel attack; an attacker can obtain keyboard interrupt timings by exhausting the pool and waiting.  Some patches by Ts’o in 2012 mitigate this and other issues.

In 11/2012, another side-channel attack was reported.  Researchers have devised a virtual machine that can extract private cryptographic keys stored on a separate virtual machine when it resides on the same piece of hardware.  The technique they used took several hours to recover the private key for a 4096-bit El Gamal-generated public key (used with GPG and elsewhere).  You can read the details from Ars Technica.

For some systems including Linux, there is also /dev/urandom, which works the same except that if the entropy pool runs out, it uses PRNG techniques and so will never block.  But the quality (“randomness”) suffers.  (This is a bit of a simplification.)  That can be taken into account by the software using it, by using a secure one-way function to reseed frequently.  But most people forget to do so when implementing their own security system.  (So don’t do that.)

You can generate strings of 10 (or any length; just change the “10”) random characters this way:

tr -cd '[:print:]' </dev/urandom \
    |od -cAn -N10 |tr -d '[:blank:]'

You can change print to alnum, to restrict the output to letters and digits only.  Use this technique to generate passwords if pwgen and/or apg aren’t available.

On some (e.g., BSD) systems, /dev/random is reserved for hardware RNG (e.g., counting cosmic ray particles per second that hit the chip).  These OSes have /dev/srandom that is similar to Linux /dev/random.  Linux does have /dev/hw_random, if such hardware is detected.

“Using pseudo-random numbers for security results in pseudo-security.”

“Random numbers should not be generated with a method chosen at random.”

Crypto Primitives and Protocols

Cryptographic systems are built from simpler pieces, known as cryptographic primitives.  Each of these is designed to have a simple function, for example the functions of encryption, integrity checking, or authentication.  Some of these primitives are discussed next.

A combination of crypto primitives is used to construct a cryptographic protocol.  Examples of such protocols include SSH and TLS.  It should be noted that even if the individual primitives are error-free, their use in a faulty protocol won’t ensure security.  There are several well-known examples of such failures, such as Wi-fi’s WEP.

Kerckhoff’s Law

Early symmetric encryption schemes had no keys or passwords!  The encryption method itself had to be kept secret.  Qu: what’s the problem with such schemes?  Ans: Since you have to share your scheme with someone else for it to be useful, it is difficult to keep secret.  If you communicate with a large group, everyone needs to have the same scheme.  Finally, it turns out to be very difficult to design a good crypto protocol, so if you have to keep yours a secret, it will not be possible to have lots of smart people study your method for flaws.

The requirements for a modern crypto protocol were first stated by Kerckhoff in 1883:

A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

This principle was restated years later by Claude Shannon, the inventor of information theory as well as the fundamentals of theoretical cryptography.  Shannon’s Maxim states the enemy knows the system.

All modern crypto follows this principle.  The scheme is public knowledge, and (hopefully) well-studied.  The security of the scheme rests with the security of a key.

Stream ciphers

Stream ciphers are a type of encryption algorithm that read a continuous stream of bits of plaintext, and produce a stream of ciphertext as output.  One very common stream cipher was RC4 (used in SSL and WEP).  Others include Salsa20 (and the related ChaCha20, which as part of the BLAKE hash, was a finalist in the NIST’s competition for an SHA successor), and Spritz (designed as a “drop-in” replacement for RC4).

RC4 is now (2016) known to be vulnerable to some attacks (e.g., the SSL/TLS “Bar Mitzvah” attack), and is no longer considered secure.  TLS 1.2 permits AES-GCM (a block cipher that is fast enough to be used instead of a stream cipher) instead of RC4, and that should be used if possible.

The key determines the encryption of the first bit of data, and the result is used as a key for the next bit.  For many stream ciphers, the sequence of key bits (known as a keystream) is simply XOR-ed with the plaintext.  With a stream cipher, if the initial part of the stream is corrupted or lost, the rest of the stream cannot be decrypted.  If the sequence of keys repeats, the length is the key period and must be long (ideally, longer than any message).

Another approach is called self-synchronizing stream ciphers.  Rather than use all previous ciphertext to determine the next bits of the keystream, only the last few (“N”) digits are used.  So an error in one bit will only affect the following N bits of output.  Thereafter the data is decryptable again.

If used incorrectly, a stream cipher is easy to crack (but then, that’s true of all cipher methods).  One common problem is bit-flipping attacks (if an attacker flips a bit in the ciphertext, the corresponding bit in the plaintext gets flipped.  Adding a MIC (message integrity code; see below) can protect against this.  Another attack is that using short keys can result in multiple messages sharing the same keystream.  By XORing two such messages together, the XOR of the two plaintexts are produced!  Statistical analysis of the result can reveal the plaintext.  Adding an initialization vector (or “IV”) or nonce effectively increases the key length.

Stream ciphers generally need less hardware and are faster than block ciphers.  This makes stream ciphers useful for network connections such as with HTTPS.

Block ciphers

Block ciphers work by transforming a fixed-size block of plain text to a block of cipher text.  When used with a variable amount of data, you need to pad any short blocks.  Some commonly used block ciphers include 3DES (replaced DES), AES (replaced 3DES), IDEA, and Twofish (replaced Blowfish which is still in wide use).

When you have more data than can fit into one block, you use some sort of mode of operation to determine what to do.  Other than ECB mode, block ciphers require an initialization vector (or IV).  This is a fixed, predetermined block of data.  It is used like salt to make sure the same key doesn’t produce the same output.  There is no need for the IV to be secret, but reusing the same IV with the same key leads to weak security (which is exactly what happened with Wi-Fi’s WEP).  (See NIST SP-800-38a (PDF) for details on block ciphers.)

·       In ECB (electronic codebook) mode, each block of plaintext is separately encrypted using the identical key and algorithm.  ECP is useful when you have a multicore computer as each core can encrypt a different block simultaneously.  While simple, it is not secure.

·       With CBC (cipher-block chaining) mode, each block of plaintext is XORed with the previous ciphertext block before being encrypted.  This mode operates much like a stream cipher in that the result of a block depends on all the previous blocks.

·       Using CFB (cipher feedback) mode is similar to using CBC mode except that only the last N blocks are used to determine the encryption of the current block.  With CFB mode, the output will still be decryptable if a block is lost.

·       OFB (output feedback) mode operates just like a self-synchronizing stream cipher.  If even a single bit is lost, only that block (and possibly the following block) will be unusable; after that the plaintext will be recoverable.

·       CTR (counter) Similar to OFB, counter mode turns a block cipher into a stream cipher.  It generates the next keystream block by encrypting successive values of a “counter”.  The counter can be any function which produces a sequence which is guaranteed not to repeat for a long time.

·       GCM (Galois/Counter Mode) This is a fast and secure variation of CTR mode that provides not only confidentiality, but data integrity as well.  (AES-GCM is recommended for use in TLS security over the RC4 stream cipher.)

There are other modes possible for block ciphers.  Except for ECB, each mode depends on the output of one or more previous blocks and thus can’t encrypt different blocks in parallel.  The different modes provide different benefits, so you would pick a mode based on the goal: confidentiality, authentication, etc.

In addition to the modes, the other important features used to distinguish block ciphers are the block size and the key length.  For example, Blowfish has a 64-bit block size and a variable key length from 32 up to 448 bits; AES has a 128-bit block size, with key sizes of 128, 192 or 256 bits.  In general, a longer key is more secure.  A too-small block size leads to weak security.  For example, the 8-byte size of DES turns out to be safe for up to only 32 GB of data max, before you must change the key.  (With such a small block size, it is reasonable to change the key every few hundred megabytes.)  But beyond a 16- or 32-byte blocks, you need not change keys for so much data that the number has no name.  Larger blocks increase the number of blocks that need padding, and that can cause issues too.  (Note that block sizes are generally multiples of 8, since that makes the code more efficient on computers.)

One Time Pad

With OTP, a string of truly random numbers (the key) is added to message, and no key is ever used twice.  (Ex: CAB-> 312, OTP=465, result=777.)  OTP is the only guaranteed totally secure encryption method possible, even in theory.  Special hardware can be used to generate the random numbers.

The problem with OTP is distributing the keys in the first place!  A practical alternative to a real OTP is to use a cryptographically secure pseudo-random number generator for the string of random numbers.  If both parties start at the same PRNG seed, then they both will generate the same sequence of key digits.  While you still need to distribute the seed value, the sequence produced is very long, so frequent replacements aren’t needed as with real OTPs.

RSA’s SecurID smart-cards (“tokens”) use this type of OTP technology, using a PIN or other user-supplied password for the PRNG seed.  This type of pseudo-OTP depends on the seed being kept secret, and it must be distributed and stored securely.

A type of OTP (see RFC-2289) is used for authentication, not so much for encryption.  It is based on an algorithm called S/Key.  By exchanging a seed value between the client and server once, the server can authenticate users.  It works this way: the seed is used with a cryptographically secure hash algorithm.  The result is called key-1.  That is used with the same hash to produce key-2.  A large sequence of keys is generated this way, say up to key-1000.  The server discards all the previous keys (and the seed).

When a client needs to authenticate to some host, the host prompts the client for a key, which should be the previous key (in this case, key-999).  If the client supplies the correct key, its hash will be key-1000 and the host knows the client knows the same sequence.  The host now stores key-999.  For the next authentication, the client must supply key-998.  Note the host doesn’t know the key it is asking for; every so often, a new seed must be exchanged and a new sequence generated.

Another popular type of OTP for authentication is time-based OTP (TOTP).  This is used often with multi-factor authentication schemes, such as used with Amazon, Google, and Microsoft.  (See RFC-6238.)  Special hardware, or (more commonly) a smart phone app is given a seed value when set up.  Thereafter, the device or app displays a number based on a PRNG and the current time.  The number gets updated every minute or so.  (Demo AWS MFA App from my cell phone,)

You can make your own OTP using 10-sided dice, or tiles from a Scrabble set.  Shake vigorously, record the sequence, and repeat.  Make only a single copy of the OTP!

Symmetric (or shared) Key Encryption

Modern crypto techniques use symmetric (or shared or private) key encryption.  With such a system, the details of the method used to encrypt the message do not need to remain secret.  Only a password (or key) does.  With such methods, an organization can and should feel free to have its security methods published so they can be reviewed for weaknesses.  Example of shared key encryption methods include most block ciphers such as the older DES (Digital Encryption Standard) and AES (Advanced Encryption Standard).  There are many different symmetric key encryption methods known.

Symmetric key uses the same key to encrypt and to decrypt a message, using two related methods (usually one is the reverse of the other).  Even knowing the methods an attacker can’t recover the plaintext from the ciphertext.  While some methods provide stronger or weaker encryption than others, generally the strength of the method comes from the long key length.  An attacker must guess keys one at a time to attempt to recover the plaintext.  Initially DES used 56-bit (7 byte) keys.  Advances in computer technology made DES obsolete, as messages can easily be cracked within hours using moderately priced hardware.  Today’s keys are often 2 kibi bits long.

A problem with symmetric key systems is that of key distribution.  How do you get the key to all who need it, securely?  Major concerns such as governments can send keys via couriers, credit-card companies and banks send PINs (personal identification numbers) via U.S. mail.  But key distribution is still a problem for B2B (Business to Business) and C2B (Consumer to Business) communications.

DES and AES are block ciphers that operate on 64-bit blocks.  How longer messages are encrypted depends on the mode (discussed earlier).

Public key (Asymmetric) Encryption

A breakthrough was made in the 1970s.  It was realized that an encryption method could have two keys, mathematically related in such a way that you could use one to encrypt a message and the other to decrypt it.  (This is the opposite of shared key: rather than one key and two (related) methods, public key uses two (related) keys and a single method for both encrypting/decrypting.)  Even knowing the method and one of the keys, it is computationally infeasible for an attacker to determine the other key!

Special methods are used to generate a pair of keys, using large random numbers to do so.  The key length used is generally much, much longer than with shared keys.

What this means is that one of the pair of keys can be made public (say posted on a website).  To send a secret message, anyone can use that key.  However even if an attacker then intercepts the encrypted message they can’t read it.  It takes the other key (the private one that I keep secret) to decrypt the message.

Plaintext + key1 = ciphertext, ciphertext + key2 = plaintext

The private key is usually kept in a file that is encrypted using a shared-key system such as AES, and a passphrase the user must remember (or keep written down in a secure location).  SSH uses public key encryption.

RSA  Three mathematicians named Rivest, Shamir, and Adleman worked out a practical method for this, formed a company (RSA, Inc.), patented the method (U.S. only), and sold licenses until the patent expired in 2000.  Their method is called RSA.  This has been refined over the years and is now an Internet standard, used for HTTPS, SSL, SSH, etc.  RSA Inc. was purchased by EMC, Inc. in 2006.

A popular example of RSA is PGP (Pretty Good Privacy), which has an interesting history; but for now all you need to know about it is that GPG or GnuPG (GNU Privacy Guard) is the modern version of this and is compatible.

Public key distribution (mostly) solves the key distribution problem of symmetric key encryption.  For two parties to communicate securely, each can send the other (using insecure means) their public key.  (Exchanging keys securely is not as easy as it may seem!  Complex protocols for key exchange have been developed.)  Public key systems are commonly used for HTTPS, SSL/TLS, SSH, wireless communications, and other popular protocols.

While RSA is still considered strong, as usual a poor implementation of it can have vulnerabilities.  One such vulnerability in a popular hardware chip that generates RSA keys was found in 2017.  The flaw allows the calculation of the private key from the public key!  [See the story in Ars Technica.]  The researchers have provided a tool you can use to check your keys at keychest.net/roca.

SSL (secure socket layer) protocol was originally developed by Netscape to provide CIA security for web browser sessions.  Version 1.0 was never publicly released; version 2.0 was released in February 1995 but contained a number of security flaws which ultimately led to SSL version 3.0, which was released in 1996.  Although well documented and popular, SSL was never documented as an RFC.  (Mention EFF’s HTTPS Everywhere Firefox add-on.)

TLS (transport layer security) is the IETF-blessed version of SSL.  It has been through four major versions.  TLS 1.0 was first defined in RFC-2246 in January 1999 as an upgrade to SSL Version 3.0.  As stated in the RFC, “the differences between this protocol and SSL 3.0 are not dramatic, but they are significant enough that TLS 1.0 and SSL 3.0 do not interoperate.”  TLS 1.1 was released in April 2006 and is documented in RFC 4346.  It contains several security enhancements.  TLS 1.2 (sometimes mistakenly referred to as TLS 2.0) was released in August 2008 and is documented in RFC-5246.  This version added several improvements, including AES cipher suite for encryption.

TLS is not one protocol; rather it is a collection of four types of security algorithms.  (So are other security protocols, such as SSH.)  For each type, a choice of specific crypto algorithms is possible.  The client and server negotiate which specific algorithms to use for each type, and other parameters.  If they can’t agree, the connection fails.  Initially, the client sends a hello packet, which includes the algorithms it will accept (and they are listed in preference order).  The server responds with a hello packet too, indicating which four protocols it agrees to use.

The four types of algorithms are

·       Key exchange (used to generate a shared session key)

·       Authentication (using public-key methods such as RSA/PKI)

·       Encryption (symmetric, used with session key)

·       Integrity checking (a MAC is used)

(Other parameters are needed as well, such as the order of encrypting and integrity checking.)  Not all combinations are possible.  Not all algorithms or combinations included in the suite are considered secure anymore.  As a public service, the IANA mantains a list of the cipher suite combinations currently considered secure (although the list also shows older mechanisms and combinations now known to be insecure, so be careful).

TLS 1.3 was finalized 8/2018, and already implemented in popular web browsers.  It is a major rewrite with many improvements:  All older and vulnerable crypto has been removed.  Unlike TLS 1.2, you cannot enable RC4, MD5, or SHA-1 even if you wanted to.  Only crypto that provides forward secrecy is allowed.  With fewer choices the configuration is easier for sys admins.  More of the handshake is encrypted.

Additional changes to the protocol have sped up TLS protocol significantly.  For example, since SSL 1.0, two round-trip packet exchanges were required for the “handshake”.  Now, only one round-trip is needed.  Also, the session key can be cached on the server for a short while, so if the same client connects again in the near future, the first packet of the handshake can also contain data encrypted with the old session key.  (That violates forward secrecy for that one packet, and they put restrictions in, for example, no HTTP PUT or other such methods that modify data are allowed in that packet.)  All following packets naturally use the new session key.

Keep in mind, as a system admin you need to be aware which crypto methods are available, which are in use, and which are considered secure.  Different systems have different ways to configure the crypto libraries.

Usually the sys admin can configure a server to refuse to use certain algorithms.  OpenSSL itself can list default choices and preference ordering; Apache 2.4 and newer uses that by default.  (Show conf.d/wpollock.conf for old SSL choice directive; there’s newer syntax now.)  Use “openssl ciphers -v [ALL]” to see the default combinations, in the preference order.  (“ALL” shows all supported combinations; on Fedora 27, there are 140 of them.)  See man page for ciphers(1) for details.

NSS is less configurable; it ships with few tools and no man pages.  Apparently, you need to rebuild the source code to change the default algorithms used.  You can set the environment variable NSS_HASH_ALG_SUPPORT to a semicolon-separated list of algorithms to support, for example: NSS_HASH_ALG_SUPPORT= (See the NSS source code for a current list and defaults.  An older list can be found here.

Digital Signatures  A neat application of public key systems is for digital signatures.  A digital signature is an encryption of a message with the private key.

This means anyone with the matching public key can decrypt the message, but since nobody but the owner has access to the private key, a successful decryption proves the messages was sent by the owner of that pair of keys.  For security, a digitally signed message is usually further encrypted with the recipient’s public key, so only they can read it.  There are other ways to provide digital signatures, for instance DSS (Digital Signature Standard).

Digital signatures are an excellent way to prevent all sorts of attacks.  By putting a timestamp in the message, an old signed message can’t be sent later as “new”.  This prevents a replay attack.

Since only the sender has access to the private key, the sender can’t later claim they never sent the message.  This is called non-repudiation.  In some states, a digitally signed message or file has the legal status of one with a holographic signature.

Because of the importance of keeping the private key secure, it is often stored in a file that is itself encrypted with AES or some other symmetric key encryption method.  To use the private key, the owner must enter in a password or passphrase (on the local system, not across a network).  This way, even a virus-infected host can’t send private keys to an attacker.  (However, a human must enter the key for each use.  Even if a program such as a mail server only read the key once (and kept a copy in RAM), you couldn’t have an unattended reboot!  Often, the key is kept in a root-owned file with mode 400, unencrypted, for convenience.)

Elliptic-Curve Cryptography (ECC) is the successor to RSA for public key cryptographic technology (used for Diffie-Hellman key exchange).  ECC can offer equivalent security to RSA with substantially smaller key sizes.  (See entropy and key length below.)  This makes ECC software stronger, faster, and more efficient in use of memory and bandwidth, than RSA.

First described in 1985 by Victor Miller (IBM) and Neil Koblitz (University of Washington), and immediately placed in the public domain, ECC is based on finding the discrete logarithm of a random elliptic curve, from a public base point, is computationally infeasible.  The larger the curve, the harder this becomes.

The IEEE P1363 standard group standardized elliptic-curve cryptography in the late 1990s, including a stringent list of security criteria for elliptic curves.  NIST used the IEEE P1363 criteria to select fifteen specific elliptic curves at five different security levels.  In 2005, the NSA issued a “Suite B” standard, recommending the NIST elliptic curves (at two specific security levels) for all public-key cryptography, and withdrawing previous recommendations of RSA.  Some specific types of elliptic-curve cryptography are patented, but many others (e.g., Curve25519) are not.  Sun, OpenSSL, Firefox, MS, Red Hat and others have endorsed this newer method, making it the likely standard for the Internet 2.0.

ECC is used much like RSA: given a user’s 32-byte secret key, Curve25519 computes the user’s 32-byte public key.  Given the user’s 32-byte secret key and another user’s 32-byte public key, Curve25519 computes a 32-byte secret shared by the two users.  This secret can then be used to authenticate and encrypt messages (usually just a symmetric key, subsequently used) between the two users.

See Ars Technica for a really good article on ECC.

Key Exchange

Public key methods are great, but very slow!  Too slow and CPU intensive to be used in the obvious way, to encrypt long messages and TCP/IP sessions, or to digitally sign files.

For digital signatures where the goal isn’t privacy of the message, the usual approach is to compute a message digest, which is a cryptographically secure form of checksum (MD5 and SHA1 are popular examples).  This digest is a comparatively small number and only that is digitally signed (encrypted with a private key).  This encrypted digest is appended to the message.  At the receiving end, the same digest is computed from the unencrypted message, the encrypted digest is decrypted, and the two are compared.  This use of a digital signature ensures message integrity.

To exchange files or messages confidentially, symmetric key encryption is much faster.  The common solution is to use public key methods to securely exchange a random huge number between two parties.  This value is used as the shared key (called the session key) for a symmetric encryption method.  Unfortunately the obvious way to exchange these keys and agree on a session key (for example, a web browser generates the session key and encrypts it with the server’s public key) is subject to several types of attacks.  To provide PFS (perfect forward secrecy), both parties must contribute to the session key.

Diffie, Hellman, and Merkle solved the problem of key exchange in the 1970s.   Diffie-Hellman key exchange is used today to establish a confidential communication session with SSH, SSL/TLS and other systems.  (Show D-H Key Exchange resource.)  This scheme can be explained this way:

1.    Alice generates a session key for use in traditional symmetric encryption (such as AES).  This session key is put into a strongbox which Alice locks with her (unshared) key.  The locked box is then sent to Bob.

2.    Bob receives the locked box, but can’t open it since he doesn’t have Alice’s key.  (Neither can any eavesdropper.)  Bob adds another lock to the box, using his (unshared key), and sends the doubly locked box back to Alice.

3.    Alice receives the locked box, and using her key, removes her lock.  The box is still securely locked with Bob’s lock.  The box is sent back to Bob.

4.    Bob uses his key to remove his lock.  Finally, Bob can read the contents of the box (the session key).

There are other key exchange protocols in use today as well (IPSEC uses IKE, Internet Key Exchange protocol), but those protocols (currently) don’t support perfect forward secrecy.  (An attack (2015) called logjam causes web and mail servers to down-grade the DH and DHE key-length to a very insecure 512 bits.  This is because of U.S. crypto export regulations of the 1990s forced that keylength to be available.  Make sure your servers do not offer the cipher suite “DHE_EXPORT”!)

One problem with this method is that the two parties must exchange several messages prior to sending the encrypted message.  Thus it is not suitable for encrypting (say) email.  Other methods, such as RSA, allow Alice to send an encrypted message to Bob if she knows Bob’s public key; no exchange is needed.  Another problem is that there is no way to authenticate the person at the other end of the exchange; Alice can’t be certain Bob is receiving the session key.  Key exchange to generate a session key is used together with certificates and other algorithms, in a security scheme (such as SSH) to overcome these issues.

Secret Sharing

Shamir’s Secret Sharing is a form of secret sharing, where a secret is divided into parts.  Each participant is given one (unique) part.  Depending on how you use this, either all of the parts or just a given number of them are needed in order to reconstruct the secret.  The secret is usually a key that was used to encrypt data.  For example, Symantec has 16 employees authorized to modify PKI (SSL) certificates.  The master key to unlock the database is split into 16 parts.  It takes any three of them to obtain the key and thus unlock the data.  (From Ars Technica.)  This is easy to use: yum install ssss; man ssss.

Cryptanalysis

Traditional crypto was all too easy to break (show crypto quote puzzle copycreate.com resource; requires “flash-plugin” from Adobe, not “gnash-plugin”).  Cryptanalysis studies the encrypted text and looks for patterns.  For example the most common letter is “e” in English.  A word such as “WXY’Z” almost certainly ends in a ‘t’ or an ‘s’.  By looking for such statistical patterns, and doing some guessing about the plaintext, the method used to encrypt the message can be guessed with some accuracy, and the keys discovered in a variety of ways.  Early ciphers such as a Caesar cipher suffered from this problem.

A weak but secret method offers security against only the most casual attackers.  Early word processors used to XOR the document with the password repeatedly:  passwordpasswordpassword... XOR plaintext (show bin/xor on YborStudent).  By XORing the encrypted document with itself, the password drops out; you have the text XORed with itself (the autokey cipher), and that is usually easy to break.  Also, often you can guess a standard header such as the “From ” or “Date: ” header in an email.  By XORing the encrypted email with the standard header, the password is revealed.

Later, stronger crypto was used as the math was better understood.  These methods first compress the message to remove redundancy, then encrypt the result.  Keys are very large numbers, hundreds of digits long.  Such keys are usually stored in a password protected file so the user can get to it more easily.

Unicity Distance

The unicity distance measures the amount of ciphertext required so there is only one reasonable plaintext message.  If the text is shorter than this, the cryptanalyst can’t be certain if they cracked the message or not.  (For example, if you intercept a single letter message “G”, and you crack the encryption and get “P”, you won’t know if that is right or not.)  If the message is long enough, the odds of a random key producing a realistic message are nearly zero.  The unicity distance says how long is long enough.

The unicity distance depends on the type of data and the key length (k/r, where k is the key length in bits, and r is the redundancy factor in the data).  For 128-bit key ciphers used today, for English text (an email message body or an office document, for example), the unicity distance is only about 128/6.8 = 19 bytes.  If the data is compressed first, or is non-text, it has less redundancy and thus a higher unicity distance (maybe twice as long).  The moral is, it doesn’t take a lot of data to be able to recognize a successful cryptanalysis.

Checksums, CRCs, Message Digests, and MACs

A checksum (or just sum) is a simple way to check for errors.  The bytes (or words) of some file are added together, ignoring any overflow.  The result is a checksum.  Checksums are also used to add a check digit to numbers, to allow simple detection of valid numbers from (say) random numbers.  This is one of the protections for U.S. currency; most serial numbers won’t be valid.

There are many simple checksum algorithms in use (see Wikipedia for check-digit), including mod-10 (a.k.a. Luhn) which is used for credit cards, GS1 used for ISBN numbers, and other algorithms used for UPC (bar) codes, U.S. currency serial numbers, etc.  (Mod-10 was designed to detect transposed digits.)

A CRC (cyclic redundancy check) is an improved version of a checksum: the data is considered a very large single number, which is divided by a special value.  The remainder becomes the CRC.  Unlike a checksum, a CRC will detect transposed values and many other types of problems that would get past a checksum.  (This is used for the FCS in Ethernet, and for the cksum POSIX utility.)

Neither of these methods is useful for security applications.  It is not difficult for an attacker to cause a modified file to have the same checksum or CRC as the original.  (These methods are great for detecting data corrupted by noise, but not from an attacker.)  So, how can you protect the integrity of your data even from attackers?  You need to understand a few concepts to see how.

A hash function is any well-defined procedure or mathematical function that converts a possibly large amount of data (a message) into a small datum (the hash, hash value, or hash code).  The hash is usually a single integer with the property that any change to the message will change the hash.  (A hash is often used to speed database lookups, not just for security.)

A one-way function is a math function that is easy to compute in one way, but the inverse is difficult.  For example, multiplying two numbers is easy, but factoring a large number into its factors is hard.  (This is the math behind RSA).

A cryptographic hash is a one-way function that supposedly can’t be easily reversed (i.e., you can’t find the original message just given the hash), can’t be easily faked (i.e., you can’t easily change the message without also changing the hash), and can’t be easily spoofed (i.e., you can’t easily find a second message with the same hash as another message).  Some better-known cryptographic hash functions include MD5 and SHA-1.

A [message] digest and [secure] hash [code] (a.k.a. “MDs”) is usually used instead of checksum or CRC for security applications.  Also called a fingerprint, file signature, or message integrity code (MIC).  These have the same overall purpose as a checksum or CRC but the algorithms used makes it very difficult for an attacker to predict what changes are needed to cause a particular result.

The NIST phased out acceptance of MD5 and SHA-1 in 2010.  Instead it recommends using to SHA2 or another more secure algorithm (see U.S. Gov. standard FIPS-180-2).  Many new algorithms were considered for “SHA3”.  The dozens of algorithms submitted to NIST for this open competition (started in 2007) was down to 14 finalists in 2010.  In 12/2010 NIST selected five finalists, including BLAKE (devised by a team led by Jean-Philippe Aumasson of the Swiss company Nagravision, and Skein, which is the work of computer security expert and blogger Bruce Schneier) and the Keccak algorithm (from a team led by STMicroelectronics’ Guido Bertoni, and which uses a novel idea called sponge hash construction to produce a final string of 1s and 0s).

MACs (message authentication code) differ from MDs in that MACs also use a shared key to compute the digest; both the sender and receiver must share this key to generate (sender) and validate (receiver) the MAC.  A MAC that uses some hashing algorithm (usually MD5) is called an HMAC.  MACs are useful for authentication and integrity: When A sends B a message, B can compute the MAC using the secret key and ensure it was sent by A, and wasn’t altered.  MACs can also be used to protect data stored in a DB.  A stores the MAC with the data; when A later retrieves the data, A can be certain the data wasn’t altered since it was stored.

HMAC-MD5 is still used, for example with IPsec, but is slow to compute and not as secure as once thought (MD5 being weaker).  There are ways to create MACs using different hashing algorithms or even using encryption algorithms to generate a hash.  HMAC-AES and GMAC are becoming used; GMAC can be computed using parallel CPUs making it very fast.

(Comparing the security of these methods is discussed below.)

Trust: Web of Trust and PKI (PKI is discussed in detail, later in the course)

A further problem is one of trust:  How do you know that the email you got from me, containing my key, is really from me and not some impostor?  This problem is hard!

One insight is that a trusted third party can digitally sign my public key, before I publish it.  A user wanting Wayne Pollock’s real public key can find it on the Internet, verify the trusted third party’s signature, and thus be assured that this key is really Wayne Pollock’s.  Of course, you need to “know” the third party’s public key, and you must “know” that key really belongs to the trusted third party.

Key signing leads to the concept of a certificate.  A person’s (or website’s) certificate is a document containing their identifying information (name, address) and their public key.  That in turn is combined with the issuer’s (the trusted third party) identity, the period of validity of the certificate, and (usually) a URL where the issuer’s public key can be found.  Then the certificate is digitally signed.  Of course, unless you know the issuer’s public key is genuine you still have problems.

The most common type of certificate in use today is the X.509 compatible one.  This standard specified what data is required (and what is optional, the format of the certificate, and other information.

X.509 certificates are encoded when stored in a file, in a variety of formats (.pem, .crt, ...).  Your computer has one (or more) databases, or stores, of imported certificates.

The GPG/PGP solution is called the web of trust.  The idea is to have your public key signed by as many people as possible.  Then if someone else wants to use my key, they only need to personally trust any one of the signers.  If they don’t know any of them, too bad!  Some popular keys are signed a thousand times, making key exchange difficult.

The U.S. government solution was (once) that they would be the trusted third party.  Anyone could have them generate a public key signed by them, who would hold all keys including the private ones in escrow, an authoritative database of keys (so they could wiretap).  This scheme did not prove popular.

A third solution is used today, a global public key infrastructure (PKI).  This scheme has certificate authorities (CAs) that everyone knows the public keys for, and everyone trusts.  PKI defines different types of certificates, such as personal and website certificates.  (You can’t use the wrong type).  PKI includes mecha­nisms to allow you or the issuer to revoke a certificate (before its expiration date).

A hierarchy of CAs is possible, with one CA signing the CA certificate on another.  Ultimately, you only need to trust a few root CAs.

To use PKI, you generate you own pair of keys, and create a key signing request (KSR) file.  That document contains your information plus your public key.  You then submit the KSR to your CA of choice, who typically will charge money.  The CA is then supposed to do a check to confirm your identity, and generates the certificate for you.

The certificates for some well-known root CAs are included in most web browsers (show).  Also, anyone can become a CA, only if you do, your CA certificate isn’t signed by another CA already part of PKI; so it won’t (or shouldn’t) be trusted by the world at large.  Still, organizations such as HCC could become their own CA, to issue certificates to students, to enable secure emails with faculty and staff, to allow secure registration, etc.

To become your own CA, you generate a pair of keys, create a CA signing request, and then self-sign the certificate (I trust myself.)  Now you have a valid (but untrusted) CA certificate, which can in turn be used to sign personal and website certificates.

PKI has stalled.  The various companies involved can’t seem to move forward.  What we have now is a few companies such as VeriSign (now owned by Symantec), Thawte, Equifax, and others whose CA certificate is well-known enough to be considered trusted.

PKI Alternatives

Convergence    Given all the problems with PKI and certificate authorities (CAs), some work on a replacement scheme has been done.  The current (2013) contender for CA replacement is called Convergence.  Essentially, this replaces CAs selected by webmasters, with notaries selected by the end user.  A problem is, how can notaries make money?  You can read the Convergence Wikipedia article, and download a Firefox add-on from convergence.io.

Pairing-based Crypto         In 1984, Shamir imagined a public-key encryption scheme where any publicly-known string (e.g. someone’s email address) could be used as a public key.  In this scheme, the corresponding private key is delivered to the proper owner of this string (e.g. the recipient of the email address) by a trusted private key generator.  This key generator must verify the user’s identity before delivering a private key, of course, though this verification is essentially the same as that required for issuing a certificate in a typical Public Key Infrastructure (PKI).  Thus, an Identity-Based Encryption Scheme enables the deployment of a public-key cryptosystem without the prior setup of a PKI: a user proves his identity in a lazy way, only once he needs his private key to decrypt a message sent to him.

In 2001, Boneh and Franklin devised the first practical implementation of an Identity-Based Encryption scheme.  Their approach uses bilinear maps [pairings].

Pairings were first introduced in cryptography as a tool to attack the discrete-log problem on certain elliptic curves.  Pairing is a mapping of numbers from one group (mathematical system) to another, simpler one.  The solving (verifying) of equivalent keys in the simpler group can be 1,000 or more times faster than trying to crack the keys in the original group.  There are several well-known pairings available.

Pairings have since found other applications in the construction of cryptographic systems.  Many problems can only be solved using pairings, such as collusion-resistant broadcast encryption (and traitor tracing with short keys), 3-way Diffie-Hellman, and short signatures.

Steganography

While cryptography can protect the content of a message, it can’t hide the fact that a message was sent (and on the Internet, some further information is available in the packet headers).  Sometimes the fact that the U.S. president is sending messages to the premier of Russia is too much information to let out.  In military terms, sending many messages to someone or someplace can let the enemy know their communications has been tapped (passive wiretapping, or man-in-the-middle) or something significant will soon be occurring.  (Traffic analysis or signal intelligence, a.k.a. SIGINT.)

To prevent such consequences, even the fact that a message was sent must be kept confidential.  Steganography is the art of concealing information, typically messages, so nobody even suspects a message was sent.  (Literally, it means concealed writing.)  Often other techniques are used too, such as cover stories and fake messages.  (Coca-Cola likely does this to prevent anyone from learning the formula for Coke, by studying shipping orders from their suppliers over time.)

Illicit uses of steganography include smuggling (e.g., the plans for a new cell phone), terrorist communications (e.g., the details of a bomb attack on a commuter rail line), and child pornography.

Historically, steganography has been common since at least 500 B.C.E., when a slave had a message tattooed on his head, and by the time he arrived at his destination, his hair had regrown.  The slave carried an innocuous message but the real message instigated an Ionian revolt against the Persians that lasted 50 years.  Other forms of steganography include when Demaratus sent a warning about a forthcoming attack to Greece, by writing it directly on the wooden backing of a wax tablet before applying its beeswax surface.  (Wax tablets were in common use then as reusable writing surfaces, sometimes used for shorthand.)

Steganography has other uses as well.  By concealing some information in a document, you will be able to show that someone copied it.  This was done with logarithm and other tables of data (usually, by putting deliberate errors in the least significant digit of certain entries).  More modern examples include fake encyclopedia entries, or a fake addition to some real entry.  This technique was recently used by Google to determine that Bing was “stealing” some of its search results.

Examples that are more modern include:

·       messages written in Morse code on knitting yarn and then knitted into a piece of clothing worn by a courier; messages written on the back of postage stamps;

·       microdots (pages of text or images photographically reduces to the size of a period, and then placed on a period in some message, or inserted inside a postcode (between layers of the paper);

·       all sorts of invisible inks (e.g., writing a message with vinegar on the shell of an egg causes the message to show on the hard-boiled egg inside)

·       watermarking (e.g., to trace press leaks the British PM Margaret Thatcher had word processors modified to encode the users’ IDs in the spacing between words in justified paragraphs);

·       digital  (or network) steganography, when messages are concealed in unused IP header fields, in the palette of GIF or JPEG images, or in specially crafted “noise” or “dropped” UDP packets used in VoIP such as Skype.

With a bit of effort, many digital covert channels can be rendered unusable.  For example, FreeBSD includes “packet scrubbing” in its network code that zeroes all unused bits in an IP packet.  Similar scrubbing can be used to cleanse GIF or other file types of any unused fields that could hold secret information.

Garage Door Openers 

(From arstechnica.com/tech-policy/news/2009/12/what-is-drm-doing-in-my-garage.ars, 12/17/2009)

Early designs for Garage Door Openers (GDOs) were simple: a remote opener would transmit an RF signal and the receiver would open the door.  This scheme is vulnerable to someone nearby recording the signal and using it later (a replay attack called code grabbing in this case).  However not one case of code grabbing has ever been reported (2009).  But using a fixed code does have problems: pranksters can drive by broadcasting a signal and hope it operates someone’s GDO.  Another problem is that low-flying aircraft send RF signals that confuse GDOs and open the doors.

Chamberlain (large U.S. maker of GDOs) now uses a two part rolling code called “Security+”.  The remote sends an ID signal and a security code.  After each open the code changes (think pseudorandom number generator sequence).  This scheme was defeated by Canadian company Skylink, who sells cheap replacement remotes.  By analyzing Security+ they found a weakness:  The Skylink universal remote opens Chamberlain garage doors by sending out the correct device identification code and then three signals, none of which have any connection to the expected security code.  The first signal, because it is incorrect, is ignored by the opener.  The second signal is also an incorrect code, but coming so close to the first, it triggers to opener to look for a third security code to follow milliseconds later, one which differs from the second code by a value of three.  If this happens, the opener triggers its own “resynchronization” sequence, resets the code windows it has been using, and (crucially) opens the garage door.

(An interesting side note is that Chamberlain tried to sue Skylink for violated the DMCA.  They were apparently laughed out of court.)

Military Drones Use Less Encryption Than Commercial DVDs

(From arstechnica.com/tech-policy/news/2009/12/predator-drones-use-less-encryption-than-your-tv.ars 12/17/2009)

Militants there have been intercepting U.S. Predator drone video feeds using laptops and a $30 piece of Russian software.  The Wall Street Journal reports (December 18, 2009): Insurgents Hack U.S. Drones; $26 Software Is Used to Breach Key Weapons in Iraq.  The military didn’t bother to encrypt the video feed!

Of course this is negligent, but keep in mind these drones rarely fly over our own positions, only the enemy’s.  And they already know what they’re doing, so the problem is really only in alerting the enemy of the surveillance.  But of course that makes the whole program much less effective.

(“It’s not a problem,” say leading experts, “we’ll just put an anti-pirating warning on the video and we’re sure they’ll be so scared that they will instantly stop the interception.  Who’s not terrified of the DCMA?  If they try to illegally download video the RIAA will sue them!” :-)

Lecture 6 — OpenID and OAuth (single sign-on across entire Internet)

Identity Systems

There are several identity schemes being developed, none widely supported yet.  Several of these schemes use SAML (the OASIS Security Assertion Markup Language).  Cat Okita defines identity (“(digital) Identity 2.0”, ;login: (Usenix) V32#5 Oct. 2007, pp. 13–20) as:

1.    Verifiable – My statements/assertions about my identity can be verified.

2.    Minimal – Only tell others the least they need to know to verify my identity.  This helps preserve privacy.

3.    Unlinkable – The different parties I’ve proven my identity to cannot link together the information each has about me, to get around the minimal requirement.

4.    Usable – The identity system must be useable by average persons, or it is useless.

These are based on Kim Cameron (of Microsoft) Laws of Identity (www.identityblog.com/?p=352), which were restated by Ann Cavoukian (www.ipc.on.ca/images/Resources/up-7laws_whitepaper.pdf) as: (1) Personal control and consent, (2) Minimal disclosure, (3) Need to know access only (justifiable parties), (4) Protection and accountability (directed identity), (5) Minimal surveillance, (6) Understandable by (average) humans, and (7) Consistent experience.  The major schemes today include:

·       Microsoft CardSpace (msdn.microsoft.com/en-us/library/aa480189.aspx) – Not unlinkable.

·       Kantara Initiative (KantaraInitiative.org), formerly the Liberty Alliance (www.projectliberty.org) – Verifiable but not currently minimal or unlinkable.  Uses SAML.

·       OpenID (openid.net) – Not verifiable or unlinkable.  Used by Web 2.0.

·       OAuth (oauth.net) - Supported by Facebook and other popular sites.  Not unlinkable.

·       Shibboleth (shibboleth.internet2.edu) and Pubcookie (pubcookie.org) – Not unlinkable, uses SAML, common in academia.

·       FreeIPA (FreeIPA.org) is supported by Red Hat (and Linux in general) for single sign-on.  It is integrated into SSSD.

·       Persona (Persona.org) is Mozilla’s re-worked BrowserID system.  It is notable for using public-key and X.509 certificate to prove your ownership of some email address, generated quick as needed.  (See this overview.)

Identity Management and Single Sign-on

The ability to manage multiple user identities is a key technology of modern, secure systems.  Identity management (“IdM”) means managing users and roles, their attributes and authority, and authentication data.  A single database is used, just as for single sign-on (often LDAP).  IdM systems are designed to support AAA (especially accountability and auditing requirements), and to support compliance with various laws and regulations.  This is known as governance, risk, and compliance (GRC).  (See the Open Group’s governance framework, which is also an ISO standard.)

IdM is a marketing term more than a technical one, and as such doesn’t have a firm definition (Hatchi-ID.com has a good definition).  In general, any IdM must support:

·       Globally unique (to the organization) IDs that will work on any system (e.g., Windows, Mac, Chrome, *nix),

·       the ability to create, modify, and delete user accounts,

·       an easy to use user interface (a secure web interface works well),

·       the ability to manage and report on user accounts, such as the ability to see all users with certain privileges, or all the privileges of a given user, and

·       automatic processing, such as password expiration and resetting (often, via email; see netid.hccfl.edu for example), automatic approvals prior to granting of access, and automatic processing of onboarding and termination of employees.

There are many commercial IdM systems, including from Oracle, BMC, and many others.  (See the Sun Buyers Guide to Identity Management paper.)

All IdMs support single sign-on, or SSO (or ESSO for Enterprise SSO).  In an SSO system, not only is there a single database for accounts, but logging in to any system in the organization provides credentials so a user doesn’t need to sign in to each system.  (The credentials last only to the end of the current session.)  This is accomplished by setting up trust relationships.  Kerberos is often used for SSO, but there are many more polished SSO products that can be purchased.

OpenID

OpenID (openid.net) is an attempt to solve the problem of having (and maintaining) multiple accounts when using the Internet, by providing a single authentication service (SSO, but not just for any one organization) for a user.  The idea is that you only need a username and password on your chosen OpenID server.  Other websites will trust the OpenID server to say if you are you or an impostor.

When a user wants to log in to any web site, instead of submitting a username and password the user submits an OpenID URL that he owns.  This URL points to the user’s OpenID authentication server.  The web site uses this URL to find the user’s authentication server and asks it, “This user claims he owns this URL.  The URL says you are in charge of authenticating this fact, so tell me, can he access my site?”

The authentication server then prompts the user to log in using the IP address and port number of your already open web browser window (to the “store”), and will then ask the user whether he wishes to be authenticated to the external site.  If the user confirms that, yes she does, then the authentication server will notify the external site that the user is authenticated.  Authentication servers such as RADIUS aren’t designed for such Internet single sign-on, so OpenID usually requires some non-standard authentication server to be deployed.

OpenID is becoming popular (2014).  It is used by several major players such as Google, Facebook, and Yahoo (announced but not yet deployed, 3/2014).  The reason is easy to see: companies that offer such services are all in the web advertising field; knowing what people are doing on competitor’s sites gives them valuable data.

OAuth

A promising protocol that is catching on (it’s supported by Facebook, Twitter, Google, and others) is called OAuth (OAuth.net).  OAuth is intended to be a simple, secure way to authenticate users without exposing their secret credentials to anyone who shouldn’t have access to them.  It also enables users to grant third-party access to their web resources without sharing their passwords, allowing third parties to create applications that can access your Google, Facebook, Twitter, GitHub, and other sites’ data.

OAuth is best thought of as a delegation protocol, more than an authentication protocol.  It is designed to delegate limited access to some of a given user’s resources, such as their Google calendars (but not their email).  The three actors are: the Resource Owner (a service such as Google that manages resources, such as calendars, on behalf of users), Client applications (some server app that wants to access a user’s resources from a Resource Owner), and a User (the human who wants to use the client app and to allow them limited access to his or her resources managed by some Resource Owner).

A user access some client app.  The client app makes an OAuth request to the resource owner, either providing the user’s credentials (twitter works this way) or has the resource owner authenticate the user directly (PayPal works this way).  If authenticated, the client app then asks the resource owner for an access token for a specific resource.  If that access is allowed by the user, a token is generated and returned to the client app.  Finally, the client app can make requests to the resource owner, passing the token as proof that it is allowed.

OAuth was started in 2006 by Blaine Cook, who was working on an OpenID implementation for Twitter.  While working on it, Blaine realized that the Twitter API couldn’t deal with a user who had authenticated with an OpenID.  He got in touch with Chris Messina in order to find a way to use OpenID with the Twitter API.  After several conversations with a few other people later, OAuth was born.  In December of that year, OAuth 1.0 was finalized.  A minor revision (OAuth 1.0 Revision A) was published in June 2008 to fix a security hole.  In April 2010, OAuth 1.0 was published as RFC 5849.  Since August 31, 2010, all third-party Twitter applications are required to use OAuth.

To have your web application (the consumer) allow users to sign in (authenticate) using OAuth, you first decide which OAuth providers to allow.  It is the provider that actually authenticates the user.  (Naturally, they need to agree to be your provider; you can’t just use them.)  All communications between the consumer and provider uses HTTPS (that is, it is all encrypted).

Each consumer must register with each provider it will use.  The providers will provide a user name and a password to identify each consumer.

To authenticate a user using OAuth, your application sends a message to the provider containing your ID, the user’s ID, and the “scope” of the request.  The consumer redirects the user to the provider’s website.  The user signs on to the provider’s site, and they will be informed of the scope of the data the application requested from the provider.  (For example, if using Facebook, the scope could request access to any Facebook data such as your inbox.)  Each consumer should only request the smallest scope needed, of course.

When the user is done authenticating, the provider will send an HTTP request packet to your application, indicating the user’s name and a nonce value.  Your application must respond to that request, including the application ID, password, and that nonce value.  The provider then makes one last HTTP request, containing the result of the user’s authentication attempt.

Qu: Notice how the provider only makes HTTP requests to your application.  Why do you think that is?

Qu: What is the privacy implication of having (say) Google as a provider?  Ans: The provider will know the name of each user logging into your application, and when (and from where) they access the client application (and what data was requested).

System Security Services Daemon (SSSD)

Single sign-on does have a problem when the network is down, or if a laptop or other mobile device can’t access the authentication server.  (Fedora) Linux now provides SSSD, System Security Services Daemon.  This provides offline caching for networked-users’ credentials, IPv6 support, and many of the features of other frameworks, including “plug-ins” for different identity providers (i.e., user databases), called domains.  Authentication through the SSSD will potentially allow LDAP, NIS, Samba, and FreeIPA services to provide an offline mode, to ease the use of centrally managing laptop users.  Some SSSD resources include the RHEL 6 Essentials, RHEL 6 Deployment Guide, and the Fedora 18 Sys Adm Guide.

SSSD was designed to be a new backend usable in nsswitch.conf (the name service switch, NSS) for use by traditional Unix authentication system calls (as done by the pam_unix module), but it can be used directly using the pam_sss module without requiring any changes to existing PAMified daemons.  Another feature of SSSD is support for multiple databases (domains).  With just NSS, you can only use one user DB per type (e.g., one LDAP db, or one Kerberos DB).  Multiple domains also handles the situation of similar usernames in different domains; you can use a fully-qualified username to distinguish which user.  SSSD provides extra functionality if using IPA version 2 as a back-end, such as dDNS support (since roaming but authenticated clients can have their IP address change).

SSSD supports older services such as NIS, common ones such as LDAP, as well as Windows Active Directory (Kerberos 5) and FreeIPA.  SSSD provides caching of credentials, supporting laptop users, or data centers with a temporary LDAP or Kerberos server outage.  Finally, SSSD includes access control similar to PAM, allowing you to lock out users (from login, not from running cron or receiving email).  SSSD includes other features as well, but this list is long enough!

If using SSSD to implement single sign-on, you won’t want to add users with the useradd utility that comes with the shadow suite (it won’t add the user to the correct database).  If using LDAP, you could use LDAP tools to update the database; you would need to use some database specific tool.  SSSD provides some utilities that make this easier, such as sss_useradd.  As long as SSSD is configured correctly, this will add the user to the correct database.  (With proper configuration, it should even add users to local files.)

SSSD solved a problem with SELinux:  All processes that check authentication originally needed permission to call getpwnam, getpwuid, and getpwent functions.  Traditionally these function calls just read the /etc/passwd file, but with nsswitch, may need network access to read NIS, LDAP, etc. DBs.  That requires too much permission!

The solution: SSSD causes all callers of the getpw* functions to use a socket to communicate with SSSD, /var/lib/sss/nss.  Then the sssd daemon does all of the network communications for the confined applications, rather than the confined applications needing to connect directly to remote some server/port.  (Fedora added a new Boolean authlogin_nsswitch_use_ldap, along with the older allow_ypbind that allows you to turn off the remote access no longer needed when using SSSD.)

After installing SSSD, you must configure one domain before starting the daemon.

SSSD is configured from /etc/sssd/sssd.conf, an INI style file, to say what single sign-on service to use and how to contact it.  (There is a sample in /usr/share/doc/sssd*, or you can use authconfig{,-gtk} to create.)  The file consists of various sections, each of which contains a number of name = value pairs (names are also called keys).  Some keys accept multiple values separated with commas.  sssd.conf uses data types of text (no quotes required), integer and Boolean (with values of TRUE or FALSE).  Comment lines are indicated by either a ‘#’ ‘;’ in the first column.  (As is usual with INI files, the definition doesn’t state if there are spaces around the equals sign, or how values can start with white-space, or if multiple entries with the same name are allowed, or the text encoding to use.)  (Show sample sssd.conf.)

To use SSSD directly, configure the nsswitch.conf with lines like so:

passwd: files sss
group: files sss

To use it from PAM, you need to update system-auth.  Here’s a sample from the RHEL SSSD guide:

#%PAM-1.0
# This file is auto-generated.

# User changes will be destroyed the next time authconfig is run.

auth required pam_env.so

auth sufficient pam_unix.so nullok try_first_pass

auth requisite pam_succeed_if.so uid >= 500 quiet

auth sufficient pam_sss.so use_first_pass

auth required pam_deny.so

account required pam_unix.so broken_shadow

account sufficient pam_localuser.so

account sufficient pam_succeed_if.so uid < 500 quiet

account [default=bad success=ok user_unknown=ignore] pam_sss.so

account required pam_permit.so

password requisite pam_cracklib.so try_first_pass retry=3

password sufficient pam_unix.so sha512 shadow nullok try_first_pass \

  use_authtok

password sufficient pam_sss.so use_authtok

password required pam_deny.so

session required pam_mkhomedir.so umask=0022 skel=/etc/skel/

session optional pam_keyinit.so revoke

session required pam_limits.so

session [success=1 default=ignore] pam_succeed_if.so service in crond \

  quiet use_uid

session sufficient pam_sss.so session required pam_unix.so

Lecture 7 — Managing System Security

Setup security

System security starts with determining the purpose of the host.  Next you determine what data on the server must be protected, and to what level.  This means examining the data on the server against regulations such as SOX and HIPAA (and HITECH), federal and local laws, and industry standards (e.g., standard accounting practices).  You must determine what data must be protected, and what data access must be audited.  All that becomes policy, which then becomes procedures and configurations.

There are a large number of best practices that should be used to provide security regardless of the use of a server.  These are discussed below.

Physical Security

Hosts should be physically secure.  Have a secured server room, using rack mounted servers (Qu: what is the standard rack server size called?  Ans: 1U), with no removable drives or connected console.  Keep track of the keys/combination.  (And have a procedure in place if an audit trail is needed of physical accesses.)  Don’t rely on having a server in a busy public place, such as a corridor or reception area.  Sooner or later the server will be left unguarded!

Don’t violate building, fire, or safety codes in your server room.  Worry about fire-suppression systems that can damage your servers.  You need adequate power supplies and/or UPSs, and A/C systems.

Best practices for physical security (when feasible)
          [adopted from a list by Sarah Scalet, sscalet@cxo.com]

·       Have a separate building for your data center, if possible 20+ miles from organization’s HQ, and 100+ feet from any public road or other building.

·       Avoid locations near airports, power plants, chemical facilities, and anywhere near earthquake fault lines flood zones, hurricane prone areas, etc.

·       Plan for secure air handling: recirculate and/or filter the air (so smoke and other pollutants can’t get in).  For the truly paranoid use air monitors.

·       Use fire suppression systems that don’t damage your equipment.  Make sure you use plenty of smoke and heat sensors.  Use humidity/water sensors if needed.

·       Don’t put up “Data Center” signs.

·       Landscape for security: Use trees (not too close to the building) and fences to keep vehicles and pedestrians at a goodly distance.

·       Thick concrete is a cheap and effective barrier against both attackers and Mother Nature.  “Safe rooms” can be made to hold backups.  Avoid windows except as required by law (building codes).  And then use warehouse type safety glass.

·       Don’t use drop-down ceilings in security sensitive areas, which provide easy access to locked rooms by climbing on a chair.

·       Make sure all contractors, repair personnel, utility and maintenance people, etc., are accompanied by an employee at all times.

·       Use vehicle crash barriers between parking areas and building.

·       Control access to parking lot and loading docks: manned guard stations with raisable gate and tire shredders on exit roads.  Have few entry/exit points.

·       Make fire doors exit only (no outside handles), and have them sound alarms when used.

·       Have a backup source for utilities (phone, data, electrical, water).  Try to have two independent connections to two different ISPs, substations, and water mains.  Have the pipe and wire conduits enter the building from two different points.  Try for underground cables where possible.  Don’t route sensitive cables through low-security areas.

·       If warranted provide anti-explosive measures: have guards check under vehicles with mirrors, use dogs (bomb-sniffing), metal detectors, etc.  Think airport security.

·       Have plenty of security cameras, with high quality recording (else it is useless).  Have alarms to both the outside of the building and on the inside (motion sensors).

·       Use two-factor authentication.  Hand geometry, fingerprint, or (when lesser security is enough) phones, smart-cards or just plain swipe cards.

·       Have multiple (say three) authentication checkpoints between the outside and the most sensitive rooms.  Have a Guest buzzer/door-bell (a reception desk is good, but may be bypassed in a rush) in a foyer.  Separate visitor areas from employee areas.  Have the most secure areas protected with a single person at a time system (turnstile), that doesn’t block egress in the event an evacuation is required.  Have each room protected as well.

·       Cleaning secure areas can be problematic.  Using filtered air and not allowing food in those areas reduces the need for frequent cleaning.

·       Provide food, meeting, lobby, and restrooms in both the secure and unsecured areas.

Secure Installation

Install disconnected from any untrusted network.  (Pull cable from back of NIC.)  Next configure firewall and some basic security systems (PAM, TCP Wrappers, ...).

Install only what is needed!  You will need to make choices of the available software.  Often a distro vendor will make some choices for you.  For example: sendmail, Postfix, or other?  You should make your choices based on reputation for performance, reliability, configurability, security, and cost.  Sendmail is an excellent MTA but is very hard to configure correctly.  Unless you have expertise available you could easily misconfigure sendmail and cause security and/or performance problems.  So if another MTA has the features you need but is easier to install it may be a better choice.

It may pay to have a local VLAN for installs: put a KickStart/JumpStart file, the set of packages you have selected, current patches, and configuration scripts to run, all on a server in a separate VLAN.  When installing, change the switch port for the server to that VLAN and do a network install!

Once basic install is done, patch/update the kernel and configure if needed for security.

Use application white-listing.  This technique is the #1 mitigation recommended by many security agencies (including the U.S. and Australia governments).  In this technique, you use special OS features (e.g., Microsoft’s AppLocker) or third-party applications to enforce the whitelist on hardened hosts.  An excellent overview of the technique can be found at www.asd.gov.au.

Bring all packages up to date.  Update yum repositories (or whatever) first: rpmfusion, adobe-flash, ....  Note that some Fedora repositories are initially disabled when installed.

General System Hardening

Okay, you have now installed, patched, configured your system, and set up policies.  Before you are done, there are a number of sub-systems you can use and configure to generally harden any server.  These are discussed here in no particular order:

Read protect critical files.  Note this include the kernel image and some other files in /boot, configuration files in */etc (and in a few cases, */lib).  Consider making system binaries execute-only; they are often readable (and writable by the owner), which facilitates updates, patching, and (virus and other) scanning, but is less secure.

Setup a secure environment:  umask, groups (/etc/group), PATH, etc.  (See /etc/profile.d/*.sh.)  Make sure the permissions and owner/group are correct for all files.

Setup syslog.conf: Fedora <9 doesn’t support boot logs.  System logging should be the first service started and the last one stopped.  Edit init.d/syslog and change the chkconfig numbers to 00 and 99, and --del then --add syslog.

Setup printer access: What you can do for this depends on the print system used, and if the printers are local or remote.  With CUPS, the default “standalone” configuration has few security risks.  The CUPS server does not accept remote connections and only accepts shared printer information from the local subnet.  When you share printers and/or enable remote administration, you expose your system to potential unauthorized access.

You can control what remote (or local) users are permitted to enqueue print jobs for your printers.  CUPS can use several user authentication schemes, such as Basic, Digest, Kerberos, and local certificate authentication.  This is a threat for local users, since their usernames and passwords are sent in plain text.  It is best to use Basic authentication over an SSH or SSL/TLS tunnel.

CUPS also allows one to set up printer quotas, which limits how many pages and/or jobs users can request.  For example:

# lpadmin -p myprinter -o job-quota-period=604800 \
    -o job-k-limit=1024 -o job-page-limit=100

Would limit all users to no more than 100 pages or 1024 KB of data (whichever comes first), within the last 604,800 seconds ( = 1 week).

Change attacker-useful system banners! (/etc/issue{,.net}, /etc/motd).  Note motd usually needs to contain an AUP or related information!

Setup trust relationships between systems: BSD-rhosts (modern versions use Kerberos): rsh, rlogin, rcp, ..., /etc/hosts.equiv, /etc/hosts.lpd and lpd.perms (used to control printer access).  Avoid the rhosts system and use ssh/VPN instead.

Verify iptables firewall: iptables -vL; /etc/init.d/iptables status

Check default permissions of standard directories and any added user accounts (show find-world-writable script).

Check for known vulnerabilities for BIND DNS, postfix, etc., at CERT/CC, Bugtraq, and elsewhere.  Apply required patches.

Securing cron and at Services: Review:  crontab files, crontab -lre, crontab file, /var/spool/{at,cron}, /etc/*cron*, anacron (run tasks after so many minutes/hours/days/...  Unlike cron it doesn’t assume the machine is always on.).  {at,cron}.{allow,deny} (if the .allow file exists, ignore any .deny file; if neither file exists than only root has access (varies by flavor).  (Default: empty .deny file.)

Solaris: *LK* as the 1st four chars in /etc/shadow password field prevents cron or at jobs of that user from running.

Adjust login account defaults (/etc/login.defs on Linux, /etc/skel,

Setup PAM, /etc/security/time.conf.  Per-user resource limits are discussed below.

Setup SSSD, /etc/sssd/sssd.conf.

Setup ssh (/etc/ssh/sshd_config, ...) and disable telnet and FTP (discussed later).  Examine /etc/hosts.allow, /etc/hosts.deny (used for TCP Wrappers) as well (discussed later).  Note OpenSSH doesn’t use PAM by default!  Update: OpenSSH since v6.7 (sshd) no longer supports TCP Wrappers.

Setup PKI: Digital certificates are now centralized (RH) in directories under /etc/pki/.  Users performing an upgrade must relocate their digital certificates.  For example /usr/share/ssl/* has moved to /etc/pki/tls and /etc/pki/CA.

Setup firewall:  Use TCP Wrappers to control access too.  Never attach a newly installed machine to an untrusted network until after it has been hardened.

Turn off all unnecessary services:  Remove unneeded software that an intruder could use, such as a C compiler.

Run auditing / vulnerability scanners:  Periodically (or always) run system and network auditing tools, which scan your system and report (or even fix) security problems), using such tools as (see a list at sectools.org).

Some of the better-known vulnerability scanners include:

·       Bastille (bastille-unix.org) (bastille -{xc} for interactive mode (X window or Tk console), “-b” (batch mode) to use saved results (config) from previous run.  bastille -r reverts all changes.  # bastille --assessnobrowser creates an assessment report only and save it in a file.  Log of changes and reports in /var/log/Bastille.  (Always review the log; some errors may be okay.)

·       Tiger is another useful system-neutral tool.  (www.nongnu.org/tiger, but not updated since 2007).

·       Lynis is a good system-neutral tool, designed especially for compliance auditing (www.rootkit.nl/projects/lynis.html).  You can run it with:
  # lynis -c -Q

·       Fedora 10 and newer includes the Red-Hat like distro specific sectool (fedorahosted.org/sectool).  This is very good to run on Fedora.

·       Debian has a distro-specific tool called checksecurity, but it is not very complete (packages.debian.org/lenny/checksecurity).  Mandriva includes a good tool msec (wiki.mandriva.com/en/Draksec).

·       The Solaris Security Toolkit (SST), formerly known as the JumpStart Architecture and Security Scripts (JASS) toolkit, is similar to bastille and provides a way to harden and audit Solaris ≤ 10 (oracle.com/.../819-1402-10).  (Oracle doesn’t support SST for S11, but it may run anyway.  Most of the tasks it did for S10 have been “baked into” S11 Express, according to someone I asked who should know.)  (Update 11/2012: Solaris 11.1 supports OpenSCAP.)

·       Nagios (nagios.org) is “the” Unix/Linux monitoring system.  It includes a plug-in architecture, so it can be used for vulnerability scanning too.  However, it requires some effort to learn, and to set up as a vulnerability scanner.

·       Nessus (nessus.org) was “the” vulnerability scanner.  It was FLOSS, but then was closed-sourced in 2005.  While still a great scanner, is no longer free; it costs $1200 per year last time I checked.

·       OpenVAS (openvas.org) is a vulnerability scanner that was forked from the last free version of Nessus after Nessus went proprietary.  It includes over 20,000 checks, and is actively maintained.  (Version 6, released in 4/2013, includes SCAP support.)

·       chkrootkit (chkrootkit.org) was last updated in 2009; this tool is no longer considered useful.

·       rkhunter (www.rootkit.nl/projects/rootkit_hunter.html)  This is easy to use and is maintained; I run it via cron on YborStudent.  (Show report email.)

·       OSSEC (www.ossec.net) is an open source IDS.  It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting, and active response.  It runs on most OSes.

·       The Automated Security Enhancement Tool (ASET) is a set of administrative utilities for Unix that improve system security by warning of potential security problems, allowing administrators to check system files settings, both attributes (permissions, ownership, etc.) and content.  You can specify low, medium or high security levels.  This is similar to Bastille and other vulnerability scanners.

·       Nikto (www.cirt.net/nikto2) is an Open Source (GPL) web server scanner that performs comprehensive tests against web servers for multiple items, including over 3500 potentially dangerous files/CGIs, versions on over 900 servers, and version specific problems on over 250 servers.  Scan items and plugins are frequently updated and can be automatically updated (if desired).

·       Shadow Security Scanner (safety-lab.com) and Shadow Web analyzer are good commertial tools with a free trial period.  (This tool found problems on wpollock.com with some PHP scripts that Nikto missed!)

·       SiteCheck (sitecheck.sucuri.net/scanner) is a free, on-line website scanner.

·       Saint (SaintCorporation.com, a commercial vulnerability scanner with a trial version).

·       Security Blanket (trustedcs.com/SecurityBlanket/) is a well-known commercial vulnerability scanner, now owned by Raytheon.  (Sun partnered with TCS so most of the JASS checks were incorporated into this, for the Solaris version.)  It includes a 7-day trial license.

Related tools monitor log files, and either send summaries, alerts, or take some action.  Some of the tools not mentioned above include swatch, fail2ban (updates host firewall to block IPs of brute-force password attacks), splunk (probably the best, but expensive; the free version doesn’t do as much), SEC (simple event correlator).

Qu: how often should these tools run?  logwatch runs once a day by default, so it can’t help you with real-time or near real-time monitoring.  On the other hand, these tools are resource intensive tools that can definitely impact the host’s performance.  Sometimes a simpler tool, such as fail2ban, can be run frequently, and more intensive tools run only hourly (or less often).

SCAP

OpenSCAP is a set of open source libraries providing an easier path for integration of the SCAP (Security Content Automation Protocol) standards, managed by NIST (who also maintain a list of commercial, SCAP compliant tools, as well as provide some content you can download).  SCAP was created to provide a standardized approach to maintaining the security of enterprise systems, such as automatically verifying the presence of patches, checking system security configuration settings, and examining systems for signs of compromise.  It can then report problems, or use a configuration management system (OpenSCAP uses Puppet) to effect changes.

The items to be checked (called “content”) are in a standard XCCDF format, or the “raw” OVAL format (used internally with XCCDF, that lists all the various checks).  Both are XML files.  XCCDF files usually reference one or more OVAL files with the actual checks to be done, and group the various checks into “profiles”.  You can select which profile to audit with, define new ones, add, enable, disable, or remove various OVAL checks from profiles.  The result of the audit is generally presented as an HTML file.

Depending on the tool, any violations can be automatically fixed with one or another configuration management system (such as Puppet); the oscap tool can generate a Bash shell script you can run (as root).

To use OpenSCAP’s secstate utility, you need to import OVAL or XCCDF “content”.  Some OVAL content repositories can be found at www.itsecdb.com/oval/, www.redhat.com/security/data/oval/ (for RHEL), or at oval.mitre.org/repository/about/other_repositories.html.  You can find XCCDF content for Fedora at fedorahosted.org, rpm.pbone.net, rpmfind.net, from the National Vulnerability Database (Checklists are another term for SCAP content) at nvd.nist.gov, and no doubt elsewhere.

Fedora 14 and newer allow you to import the content and tools from the yum repos (secstate, openscap, openscap-utils, and openscap-content), but that may not be current.  Using the OpenSCAP utility oscap, you can also generate content.

 

secstate, the Security State Configuration Tool Example:

# yum -y install secstate
# wget https://fedorahosted.org/openscap/log/dist/fedora/scap-fedora14-xccdf.xml
# secstate import scap-fedora14-xccdf.xml  # can take a long time!
# secstate list -a -r   # can take a long time, and should show the following:
[X]OVAL File - ID: scap-fedora14-xccdf
# secstate audit
--Results--
True:           0
False:          0
Error:          0
Unknown:        0
Not Evaluated:  0
Not Applicable: 0

(Detailed results get saved in HTML, in a folder with a name such as./audit-localhost.localdomain-Wed-May-25-15_52_27-2011bEd6I3/ (show).)

Update: On Fedora 20, secstate tool can’t import any content!  Use oscap instead.

oscap (the OpenSCAP utility) Example:

This works from Fedora 14 thru Fedora 27 (at least), if you use the correct profile name:

# yum -y install openscap openscap-utils openscap-content scap-workbench

# oscap xccdf eval --profile common  \
--results /tmp/$USER-ssg-results.xml  \
--report /tmp/$USER-ssg-results.html  \
--cpe /usr/share/xml/scap/ssg/fedora/ssg-fedora-cpe-dictionary.xml  \
/usr/share/xml/scap/ssg/fedora/ssg-fedora-xccdf.xml 

# oscap xccdf generate --profile common fix \ --output xccdf-fix.sh \
/usr/share/openscap/scap-xccdf.xml

The scap-workbench utility is a GUI tool to run scans and to edit SCAP content.

To review what checks and tests are performed, you can read the XCCDF content in the /usr/share/openscap/scap-fedora14-xccdf.xml file.  You can generate an easier to read HTML document instead, like so:

$ oscap xccdf generate guide /usr/share/openscap/scap-fedora14-xccdf.xml \
    > fedor14scap.html

Evaluation of all OVAL definitions:

# oscap oval eval --results ~/oval-all.xml /usr/share/openscap/scap-oval.xml
# oscap oval generate report --output ~/oval-all.html ~/oval-allresults.xml

Evaluation of specific OVAL definition (perhaps you saw an alert from CERT):

# oscap oval eval --id oval:org.open-scap.f14:def:20055 \
    --results ~/oval-20055results.xml /usr/share/openscap/scap-oval.xml

(In all cases, running an audit can take a very long time!)

firstaidkit (the System Rescue Tool), automates simple and common system recovery tasks.  It can also be used to diagnose any sub-system for which it has a plugin available (yum install firstaidkit firstaidkit-plugins-all), making it a useful vulnerability scanner:

# firstaidkit -a --nodeps |
   awk -vflag=0 'flag {print; flag = 0}
   /ISSUE FROM/ {print; flag = 1}'

[LOCAL] ISSUE FROM Free Space
'Waiting for check on Free Space'
[LOCAL] ISSUE FROM Free Space
'No problem with Free Space'
[LOCAL] ISSUE FROM X server
'Waiting for check on X server'
[LOCAL] ISSUE FROM X server
'No problem with X server'
[LOCAL] ISSUE FROM mdadm configuration
'Waiting for check on mdadm configuration'
[LOCAL] ISSUE FROM mdadm configuration
'No problem with mdadm configuration'
[LOCAL] ISSUE FROM Grub
'No problem with Grub_DIR'
[LOCAL] ISSUE FROM Grub
'Detected Grub_IMAGE -- Bad grub stage1 image.'

STIGs (Security Technical Implementation Guidelines)

[Adapted from Jim Laurent’s blog, read on 10/4/2011.]  DISA (U.S. DoD’s Defense Information Systems Agency) owns and operates the DoD datacenters, develops a number of command and control applications, runs the DoD networks and is responsible for enforcing DoD security mandates.  The STIG checklists are a comprehensive set of requirements that system administrators are expected to follow in order to attach and maintain a system on DoD networks.  There are STIG documents for databases, firewalls, web servers and other applications, as well as general STIGs for various operating systems.  Unix/Linux SAs should be most concerned with the STIG documents for Unix/Linux systems.

The DISA STIG checklist is a public document that describes specific permissions settings, password policies, administrative record keeping and more.  In the current version, Section 3 is 546 pages long and is where all the specific requirements can be found.

There is also a collection of Security Readiness Review (SRR) scripts that automate portions of the review process to assist a system administrator in evaluating the completion of the process.  The STIGs also support SCAP standards for auditing; content is available for some systems and applications (such as for RHEL, from nvd.nist.gov).

Run system and network intrusion detection systems such as ipcop (ipcop.org).  Many auditing tools can also serve as IDS.  (IDS tools are discussed later.)

Run your malware detection software regularly.  For example:

     # rkhunter --propupd  # Do once to update the rkhunter properties file.
# rkhunter -c --pkgmgr RPM --rwo -x

 (Or set up as cron jobs.)

Console Security:  A method to force a password on all consoles during any boot is to simply modify inittab and add/edit these two lines (Old SysV init only):

d:3:initdefault:
~~:S:wait:/sbin/sulogin

(Note the double tilde.)  This forces the requirement of any legal logon to access the system even when in single user mode, preventing bypassing login security by booting in single-user mode.  For this to work, you need to protect the BIOS and boot loader with passwords, disallow booting from removable drives or remotely, and have a lock on the case.  Note BIOS security is very weak except on laptops (where it is merely weak).  Secure remote consoles as well.

There are several good lists of effective security controls.  You should consider these carefully, and implement all you can:

www.sans.org/critical-security-controls

www.counciloncybersecurity.org/critical-controls

NSA's IA top-10 Mitigation Strategies (pdf)

PCI DSS Security Control Prioritization

Lecture 8 — Managing Password Security and Database Security

Password Security

Unix/Linux systems have never stored plain text passwords on the disk.  When someone tries to login, they supply a username and a password.  The username is used to locate the encrypted version of the password.  This is often stored in the file /etc/shadow (older systems used /etc/passwd, which is publicly readable), or in some central (organization-wide, networked) database.  Although they should not be available to others, it should still be computationally infeasible to guess the password even knowing the encrypted version.

For managing a single host, the shadow suite can be used instead of a more complex SSO database.  This keeps the password hash in /etc/shadow, and only an “x” as a place-holder in /etc/passwd).  The shadow file has root-only access (show: ls -l /usr/bin/passwd) so utilities that manage passwords are often setUID to root.  Discuss the format of /etc/{passwd,shadow}.

Actually, the password is a secure message digest or secure hash, and not an encryption.  (An encrypted message is supposed to be reversible so you can recover the message.)  The phrase encrypted password is (mis-) used to refer to a hash of a password, possibly because the hash was originally calculated by the crypt library using DES.

There are a number of authentication systems available and each is configured independently of the others.  This can lead to confusion!  For example, any settings in the file /etc/login.defs or (for Solaris) /etc/default/passwd (such as for minimum password length or expiration data) only apply when using the shadow suite, not for LDAP, NIS, Kerberos, etc.  Locking passwords in the shadow file may have no effect on ssh (and its subsystems such as scp and sftp) when using keys.  PAM may or may not be used for FTP, rsh.  And so on.

There are ways to prove a user’s identity, other than supplying a password (something you know).  This include bio-metrics (something you are), and various security devices such as smart cards and dongles (something you have).  However, all these mechanisms have their vulnerabilities and drawbacks.  For all their faults, passwords are very convenient and in many contexts, offer a reasonable level of security if used correctly.  (Increasingly, two- or multi- factor authentication is used, where two of the three types of authentication must be used together.)

Change default passwords.  Default passwords are an easily and commonly exploited vulnerability.  You need to change all default passwords before any system is exposed to the Internet.  (Today the time to attack for a vulnerable system connected to the Internet is seconds to minutes.)  Don’t forget to change passwords on network devices such as routers, switches, wireless access points, etc.  New user accounts need good passwords as well.

Use password generators to pick truly strong passwords (pwgen, apg, pwqgen).

To change or set a password for the root user is a more major event than changing one for an unprivileged user.  The root passwords should be saved in a secure location.  Some Linux systems don’t ship with a root password set; not safe for a server!  Use “sudo passwd” to set the root password on those systems.

Good passwords: Actually, “good passwords” is an oxymoron.  If a password is easy to type and remember, it is vulnerable to a dictionary attack.  But some passwords are better than others.  Use the first letter of words (or better, each syllable) of a quote or poetry, plus symbols or digits: “Rar,Vab”, “Iafyds,civ1.0.”  (If at first you don’t succeed, call it version 1.0.), “"Rd"isnaip!” ("RAM disk" is not an installation procedure!)

A (longer) password composed of words is called a passphrase.  (The term passcode is sometimes used to mean either a password or passphrase.)  Usually, a passphrase is composed of legal words, preferably chosen at random, and combined with symbols or spaces.  When possible, use a mix of uppercase and lowercase letters, plus digits, punctuation marks, and other symbols.  (See below for ways to generate passphrases.)

Not all systems accept strong passwords or passphrases, particularly those with a web interface.  (Gmail permits passwords as long as 200 characters, Yahoo Mail allows 32-character passwords.  Hotmail only allows 16; for years it has accepted longer passwords, but ignored all but the first 16 characters.)

The password changing command “passwd” does a very minimal amount of checking for poor passwords.  On modern systems, PAM can be used to set a password policy.  On such systems you use arguments to the pam_pwquality.so (compatible with the older pam_cracklib.so) PAM module to force very strong passwords, or disallow (or allow) null (no) passwords.  You can even prevent users from reusing their last few passwords, or having the new password too similar to the previous one.  (Recall that the passwd command makes you enter your current password as well as a new one.)  Different systems use slightly different PAM modules for this purpose; pwquality.so is used on Fedora and Ubuntu at least.  (The libpwquality package includes a pwscore(1) utility, allowing one to check for bad passwords easily.)

Note that setting minimal password length in other files (such as /etc/default/useradd or /etc/login.defs) may have no effect, if the shadow suite isn’t used.

The purpose of picking hard to guess passwords is to prevent attacker from using lists of common passwords.  But instead of forcing users to pick complex passwords, it turns out it is sufficient just to prevent them from picking common ones.  For example, Twitter allows any password not in a list of about 400 common ones (such as “p@ssword” or “ABCD”).  There is a list of 106 passwords Blackberry forbids.  Also, an organization can track how often a given password is used, and add them to the list if too many users try to use it.  Any password, even if a dictionary word, would be allowed as long as it is rarely used.

In general, a passphrase is stronger (harder to guess) than a password, and good ones are easier to remember than good passwords.  The weak passphrases users typically employ create a major vulnerability to password cracking attacks.

Password meters are often used to indicate, when choosing a password, if the password is good, fair, or weak.  Recent work on these meters shows that most are inaccurate.  A group of CMU researchers built a new meter, using an AI (neural network) trained for this purpose, with some success.  You can download it from Github.com/cupslab/password_meter and try an on-line demo at cups.cs.cmu.edu/meter.

Measuring Strength of Crypto Primitives

How should we measure the security of a crypto primitive or a whole crypto system?  This is important since we need a way to compare different crypto mechanisms, are to express our belief in a given method’s security.

The standard answer is to determine the number of operations required to break something.  Consider the casting-out nines checksum; at worst it would take ten attempts to break the system (in this case, break would mean to alter a message but have it produce a valid checksum), where each attempt means to compute a new checksum (so, not too many operations per attempt).  The number is expressed as the number of binary digits needed, in this case, between 3 and 4 bits.  Crypto primitives are often designed to require a large number of operations to check or decrypt something, making each such attempt time consuming.  MD5 has strength of about 64 bits (depending on what you are using it for).  Originally MD5 was thought to have a strength of 128 bits, but someone found ways to reduce the number of operations needed to generate an MD5 hash, so you could then generate them much more quickly.

For confidentiality, encryption primitives require a certain (large) number of operations in order to decrypt a message without knowing the key.  With the key, only a few steps are needed.  Thus, the strength of most crypto primitives depends on the length and strength of the keys used.  Or switch to a different security primitive that requires more steps each attempt; then a smaller key gives the same security.

A good estimate of the strength needed by various crypto primitives, in order to provide a certain level of protection against various types of attackers and for various periods of time was compiled by the ECRYPT 2 project, and is available at www.keylength.com.  (Show.)

Entropy, Key Length, and Password/Passphrase Length

The strength of passwords and keys are measured in the number of random-bits they are equivalent to, also called its entropy.  It’s been estimated that normal English text only contains about 2 to 2.5 bits of entropy per character, and that the 2,000 most common words have about 10-12 bits per word.  So a four word phrase would seem to provide about 40 bits, except that most phrases aren’t selected at random, and so contain much less entropy.  About 1% of passphrases used by Amazon’s PayPhrase system users were guessed from a dictionary containing 20,656 phrases of movie titles, sports team names, and other proper nouns; that’s less than 15 bits of entropy.

While vendors and others like to talk about key length, it is the entropy that matters.  A trivial example makes this clear.  Suppose a key-generator uses this method to generate a key of some desired length:

    key = 1;
    while ( key.length < desiredKeyLength )
        append a "1" to the key;
    return key;

No matter how long the key, it has zero bits of entropy.  Recording the gender of a random person as a key has one bit of entropy.  The reason longer keys are thought to be more secure than shorter keys, is the assumption that longer keys contain more entropy.  That is generally true for passwords (where each additional character adds entropy), but not for keys.

Additionally, a poor algorithm does not use all the entropy in a key.  For example, the original HTTPS implemented in Netscape Navigator generated 128-bit keys, but the random number generator’s algorithm meant such keys had a maximum of 20 bits of entropy.  Similarly, the A5/1 algorithm used for GSM cell phone encryption uses 64-bit strong keys, but the algorithm can be cracked as easily as if it had a 30-bit key.

Microsoft (and others) used to use a user-entered password of eight characters to generate long keys of 128 bits or longer.  But the entropy of such keys is not 128, it is the entropy of the password, about 16 bits.  And in reality, it is much less than that, as humans don’t pick truly random strings for passwords.  An attacker could probably guess the correct key by trying only a few thousand passwords, so advertising the security as 128-bit keys is misleading.

If you use GPG and password protect the file containing the private key with a five-character password, does it matter that the key length is 1,024 bits?  Probably yes, since you expect attackers to intercept messages, but not to have access to the hard disk containing your private key.  If they do get a copy of your key, you have a few days to change it before they can break your password, assuming you picked a good, long passphrase.

How long should a key/password/passphrase be?

It is a common mistake to believe that longer keys are always better than shorter ones.  The point of long keys (those having lots of entropy) is to make it unfeasible to try a brute-force attack.  A good analogy I read in Bruce Schneier’s excellent book Secrets and Lies is to consider the lock on your home’s front door.  These are usually five pin tumbler locks, where each pin could be in one of ten positions, for a total of 100,000 possible keys.  If a burglar could carry around that many keys, and had to try (on average) 50,000 of them, they would need to stand at your door day and night for about three days, constantly trying the keys.  This is not feasible, and no burglar would do that.  Now, suppose I came along and offered to sell you a lock with eight pins, each of which as 12 positions.  Would you feel any more secure if you installed such a lock?  Probably not!  Once brute-force attacks are not feasible, making them even less feasible does not enhance security.  At that point, an attacker will attempt some other way to break your security.

The DES encryption standard used 56-bit keys.  In 1998, the EFF funded a machine that could brute-force attack on DES; they called the machine “DES Deep Crack” and it tried around 90 billion keys a second.  It could crack a DES encrypted message in about 4 and a half days.  A year later, a distributed grid of computers on the Internet could be used to guess 250 billion DES keys every second.  Triple-DES has 56-bit, 112-bit (two key version), or 168 (three key version) keys.

Qu: Most modern algorithms including AES use 128-bit keys.  Is that long enough?  Ans: Consider if you could build a computer a billion times as fast as Deep Crack, it would still take a million years to brute-force guess the key.

(Of course, systems may have other vulnerabilities including side-channel attacks, which make the key entropy moot as the attackers break in using other means.)

Many companies now use DomainKeys Identified Mail (DKIM) to include a digital signature and to confirm that an email originated from their domain.  These are created using RSA public keys.  However, in late 2012, it was found that companies such as Google, eBay, Yahoo, Twitter, and Amazon use 512-bit keys.  According to Harris (a security researcher), these keys can be cracked within three days in the cloud, using Amazon Web Services (AWS) at a cost of about $75.  Financial institutions such as PayPal, US Bank and HSBC reportedly use 768-bit keys; that are also considered insecure for the purpose.

To make digital signatures highly secure, keys with a minimum length of 1,024 bits must be used for RSA.  The US National Institute of Standards and Technology (NIST) in 2013 recommended a minimum length of 2,048 bits for RSA keys.

For current minimum key length recommendations, visit www.keylength.com.

The situation for passwords and passphrases is only a little different than for keys. Since passwords contain so little entropy, they can be attacked by brute-force. So making them longer makes them harder to crack. (It is doubtful a usable password or passphrase would contain so much entropy that it would not be feasible to attack this way.)

As mentioned earlier, the FIPS-112 standard requires 4 characters as a minimum password length.  The NSA recommends 12 characters.  Microsoft, who in most password-related matters adheres to NSA recommendations, recommend a minimum length of 8 characters.  As stated previously, your instructor favors strong passwords with lots of randomness (entropy), and the actual length isn’t so vital.  Still, it wouldn’t be difficult for a modern attacker to be able to easily and quickly brute-force through all possible passwords shorter than 8 characters, and longer is better.  (CERT recommends stronger passwords for administrator accounts, for example, a minimum length of 15 for root.)

It is recommended that a passphrase of at least 5 random words or 14 completely random letters be used.  For high security needs, 8 random words or 22 random characters should be employed (if you can remember them, of course).  Passphrases should be changed whenever an individual with access is no longer authorized to use the network or when a device configured to use the network is lost or compromised.

Examples of passphrases: “Sheriff town, 4 Prison keep?”, “salami-pavement, gracious Volvo”, and “Why use XKCD when, the Nerf-Herder from Tatooine is the key?”.  These are even more effective if you replace the spaces with (random) symbols.

A good tool for generating quality passphrases is pwqgen, part of the passwdqc package.  (See also pwqcheck, part of the same package.)

Password Complexity Rules

Most systems enforce some password complexity rules, in the hope that this will cause only good, string passwords to be chosen.  This rarely works.  Additionally, nearly every host and website uses different rules.  This makes a good password hard to reuse (which is a good thing), but it is annoying to most casual users, who prefer similar rules for all sites.  Still, many organizations recommend various complexity rules, including NSA, CERT, SANS, and Microsoft.

As an example of such rules, here are the default Windows Server 2008 rules.  Passwords:

·       cannot contain the user’s account name, or any part of the user’s real name longer than two characters.

·       must be six or more characters in length.

·       must contain some characters from at least three of the following four categories: uppercase (A-Z), lowercase (a-z), digit (0-9), non-alphanumeric chartacters (symbols and punctuation marks).  Note, Windows uses English for the definition of letters and digits.

Naturally, government and military requirements for passwords are stricter.

In general, for non-military-grade protection requirements it is better to take the approach of BlackBerry and Twitter, and simply ban the most commonly used passwords (which are in available word lists that hackers use).  However, enforcing some minimal password strength is a good idea by using a simple cracking tool such as pam_pwquality to check that passwords aren’t trivial to crack, before allowing their use.  (pam_pwquality is discussed elsewhere.)

Stanford University has (2014) a policy designed to make secure passwords easier.  The complexity rules for passwords decrease as the password length increases.  Passwords can be as short as eight characters, but only if they contain a mix of upper- and lower-case letters, numbers, and symbols.  The standards gradually reduce the character complexity requirements when lengths reach 12, 16, or 20 characters.  20-character or longer passwords can contain any have no complexity requirements, and could be all lower case.  In addition, passwords must pass additional checks to prevent common or weak passwords (e.g. “P@ssw0rd1”).

Password Aging

Passwords and accounts can be disabled after some time has elapsed.  This is known as aging.  Most organizations have differing rules about how long a password can be used.  Additionally, there can be rules preventing a recently used password from being used again.  Many organizations use 90 days (three months) before a user must change their password.  This is called the password’s maximum age.  FIPS-112 mandated a user change their password at least once each year.  Some people recommend (and some organizations enforce) shorter password ages: 60 days is common, while the NSA (and Microsoft) used to recommend 42 days (6 weeks).  (That was for Windows XP; they don’t seem to have specific requirements since then for Windows systems.)  For RHEL 5, NSA recommends 60 days.

In all cases, there is a minimum password age recommendation, which prevents a user from changing passwords too often.  The usual recommendation is 2 days.

Password history is the saved previously used passwords.  A system can remember the past few passwords (or actually, the hashes of those passwords).  After all, it does no good to force the user to change their password, only to allow them to reuse the same one.  NSA recommends remembers the most recent 5 passwords; Microsoft recommends remembering the last 3.

Use chage on Linux, passwd on Solaris.  Use “chage -l” to view aging info for some account.  Aging defaults for newly created logins are in /etc/login.defs on Linux, for Solaris /etc/default/login.  Aging data originally was kept in the second (password) field of the /etc/passwd file using commas as field separators.  Modern systems don’t do that, they use fields in the /etc/shadow file, or columns in DBs such as LDAP.

For each account the system holds

1.    the date the password was last changed (#days since 1969)

2.    the min # days that must pass between password changes

3.    the max # days since last change before a password expires and must be changed

4.    password expiration warning #days

5.    the # days of inactivity before an account is locked (confusingly called password inactivity)

6.    the date the account expires.

You can force a password change (by setting the date of last change to zero or 1, and setting the max # days reasonably), force users to change their passwords regularly, and prevent them from changing passwords too often.  (If min > max then users can’t change their own passwords at all.)

Use aging and PAM (pam_pwquality) to enforce password policy.  To make passwd command record old password hashes to /etc/security/opasswd (the following is all one line):

     password required pam_unix.so use_authtok md5 shadow remember=3

The pam_ pwquality hashes new passwords and checks that against your old saved passwords in opasswd.  It also tries a few hundred variations of what you type, to see if any of those match the old hashes.  It hashes what you entered, the reverse of that, that with all variations of case.  It also checks the length of the password you entered and performs some triviality checks on it:  length (strength) of what you entered, your user name, or a (small) dictionary of common passwords.  The “diffok” argument controls how many characters must be different from your previous password (remember, you had to type that in too).

(Review man pam_pwquality options.)

Password Mechanisms

Originally, only a single mechanism “crypt” was used in Unix.  Today most platforms provide several different crypto mechanisms that can be used to create password hashes.  The list of crypto mechanisms available varies by platform.  Since Unix is used in high-security environments the crypto stuff is often part of the platform’s “trusted base” and adding new mechanisms is involved (since the OS must then be re-certified).

On Solaris, the list of available crypto mechanisms is listed in the file /etc/security/crypt.conf.  The mechanism used for passwords is listed in the file /etc/security/policy.conf (the “CRYPT_DEFAULT=” line).  Some versions of Solaris use “unix” (which is really the old crypt mechanism under a different name) by default.

The mechanism used to hash new passwords (when you run the passwd command) is defined by an option in the PAM module used to set passwords (e.g., “password sufficient pam_unix.so md5 ...”).

A password hash stored in /etc/shadow (or elsewhere) could have used any mechanism, so the one used must be stored with the password.  If the first character is not a dollar sign, the old crypt mechanism is assumed.  Otherwise the first part is “$mechanism$”, where mechanism can vary depending on your system and which mechanisms are installed.

The original Unix crypt password mechanism (adapted from Robert Morris and Ken Thompson, Password Security: A case study, Published in Communications of the ACM, vol. 22, issue 11 (Nov. 1979), pages 594–597) isn’t used much today but is instructive to examine:

1.    A 12 bit random value is chosen and encoded to form a two character string from the ASCII letters and digits.  This is called the salt.  Often the current time is used to generate the salt.

2.    Next, read the first eight characters of the password (the rest is ignored).  If a shorter password is used, append the two characters of salt, then pad with zeros until 8 bytes are available.  Using 7-bit ASCII, this results in a 56 bit partial key.

3.    The salt is added to the partial key to generate the longer key that is used with the modified DES encryption used (t).

4.    A message (a string of eight bytes initialized to all zeros) is then encrypted using the key.  A modified version of DES is used that incorporates the 12 bits of salt.  This prevents using commercial DES hardware to speed the brute-force checking of keys.  The output is 64 bits (eight bytes) of encrypted data.

5.    The encrypted output is used as the input message (in place of the zeros) and the encryption is repeated, for a total of 25 DES encryption “rounds”.  (Although DES encryption is used, the original message is known and thus this is more of a hash of the password than an encryption of the all zero message.)

6.    The 64 bit result is then encoded to be all printable characters (letters, digits, periods, and slash characters only) that results in an 11 character ASCII string.

7.    The two character salt plus the 11 character encrypted text is stored in the password file.

To validate a user’s password, the entered password plus the retrieved salt value is used to generate the 11 character result.  This result is compared, byte for byte, with the copy in the password file.

Password Salt

The salt serves several purposes:

·       Without salt, if two users had the same password, the stored result would be the same.  So one user could repeatedly change their own password to guess another’s.  With salt, even when users have the same password the salt will be different, and the stored values will be different.

·       The 12 bits of salt used in crypt increases the effective length of a short password, effectively making it 4048 times has hard to guess as passwords generated without the salt.  Without salt, a “rainbow table” of all 6-character or less passwords can be generated and stored on a hard disk.  This makes it trivial to lookup a hash and recover the original password.  (Technology improvements make rainbow tables for 16 character passwords possible today.)

Adding salt to the password means generating 2-18 character password hashes instead of 0-16 character passwords.  Today, MD5 and more modern secure hash schemes use even more salt.  (Unlike the original crypt, the full password and all the salt is always used.)

·       A rainbow table is a huge lookup table used in recovering the plaintext password from a password hash.  The table takes a long time to build and a lot of storage, but once constructed it can be used to quickly lookup a plaintext password from its hash.  Salt is often used with hashed passwords to make this attack more difficult, often infeasible.

Of course, using very long passphrases / passwords (>100 characters) can be even more effective than adding salt.  But users rarely chose truly random passwords, and prefer shorter ones, so using salt is an effective solution.

·       The use of salt to modify DES makes hardware implementation difficult.  (In the 1970s, it was considered very impractical if not impossible to do so.)  This forced the use of software DES.  Having to do 25 rounds of DES in software is fast enough when validating a password, but slow enough on (1970s) hardware to make brute-force cracking of passwords impractical.

Today, implementations of DES are much faster, as is the hardware used.  A moderately priced cluster or multi-computer (SMP) can perform over 6 million DES encryptions per second.  Today’s hard disk size also means that an opponent can pre-compute all possible 13 character crypt passwords at their leisure (there are 62^13 = 200,000,000,000,000,000,000), and quickly lookup any given password.

Other terms for salt are nonce and initialization vector (IV).  Crypto experts can make a distinction between salt (a fixed value that is added to password hashes stored, and stored as plaintext) versus a cryptographic nonce (different for each use, often used with MACs and block ciphers, and not necessily chosen at random), but most use the terms interchangeably.  (A nonce is an arbitrary number, used only once: number + once = nonce.)  A non-random IV is a nonce (most algorithms require an IV to be random).  Database primary keys are nonces too.

The MD5 Mechanism:  Until recently (2012), passwords were often encrypted using MD5 (previously with crypt).  See RFC-1321.  This is actually a hash of your entire password (not just the first 8 characters, and all 8 bits in the bytes are used not just 7).

Next, this is combined with a variable amount of salt: up to eight characters selected from all letters, digits, period, and forward slash.  (If a two character salt increases the strength 4,000 times, imagine how much harder it would be to crack MD5 with a variable length salt up to eight characters!)

The result is padded with a single 1 bit followed by 0 bits, to a length of 448 bits.

The length of the password is now appended as a 64 bit number.  This results in a message exactly 512 bits (64 bytes) long.  It has been discovered that adding the message length to the message greatly strengthens the security of the hash.

Finally, the MD5 hash of this message is calculated. The resulting hash is 128 bits (16 bytes).  It is then encoded as a string of 22 characters from letters, digits, slash and period (double the old crypt length).  The salt is separated from the hash with a “$” (not needed if the salt is 8 characters long, but allowed).

You will notice a password stored in /etc/shadow begins with $mechanism$.  This indicates the mechanism (algorithm) used to hash this password.  For MD5 (“$1$”), the (up to) 8 characters following the second dollar sign compose the salt, also known as the seed.  As noted above, the salt is separated from the actual hash by a ‘$’ (unless the salt is the max length of 8 characters, in which case the dollar-sign is optional).

Following the salt is the actual hash of the password.  The result stored in the shadow file looks like this: $1$salt$hash.

As with crypt in the 1970s, MD5 for a long time was believed difficult to implement in hardware cheaply.  The huge range of hashes possible would be difficult to pre-compute and store on older disks, and might take hundreds or thousands of years of constant computing to guess on relatively available computers today.  Multi-million dollar special password cracking computers could be built that could cut the time down to mere decades.  So, how often should you change your password to be safe?  (See below for a discussion of this.)

The SHA Mechanism:  SHA-1 is a similar secure hashing algorithm.  It includes a specific 64 byte value that is added to the password and salt.  The resulting hash is 160 bits (20 bytes) long.  Stronger password hashing algorithms include SHA-2, and variants of that with longer keys: SHA-256, SHA-384, SHA-512, and others.  These are preferred for new applications because they are well studied and standardized (and required) by NIST SP-800-57.

Comparing Password Hashing Mechanisms

In June 2012, 6.5 million LinkedIn password hashes were made public.  They used SHA-1, so it didn’t matter, right?

Wrong.  Because SHA-1 uses a single iteration to generate hashes, it took security researcher Jeremi Gosney just six days to crack 90 percent of the list.  SHA1, MD5, and a variety of other algorithms widely used to protect passwords are unsuited to the job, because they were designed to generate hashes quickly using a minimal amount of computing resources.  Compare that with the original crypt algorithm, which was specifically designed to take lots of time and resources to compute.

Today, there are better hashing algorithms for passwords known, such as bcrypt, PBKDF2, scrypt, or SHA512crypt, that take lots of memory and CPU cycles to generate.

The official SHA1 specification calls for 1,448 distinct steps to calculate a hash.  By 2012, crackers had figured out how to reduce the number to 1,372, and by using special hardware instructions supported by GPUs, the number of steps needed was cut to 868.  Jens Steube (who is better known as Atom, as the pseudonymous developer of the Hashcat password-recovery program) then figured out a way to remove identical computations that are performed multiple times from the process of generating of SHA1 hashes.  That reduced the number of steps needed to 734 (meaning those LinkedIn passwords could have been recovered in only 5 days).

In 12/2012, a researcher demonstrated a cluster of five servers equipped with 25 AMD Radeon GPUs, communicating at 10 Gbps and 20 Gbps over Infiniband switched fabric.  The machine is designed to brute-force crack passwords from their hashes.  The system was able to generate 348 billion NTLM password hash checks per second; a 14-character Windows XP password hashed using LM, for example, would fall in just six minutes.  The system is less effective against stronger hashing mechanisms; it “only” generates 77 million brute force attempts per second against MD5crypt, 71,000 guesses against Bcrypt, and 364,000 guesses against SHA512crypt.  (Note, these attacks depend on the attacker obtaining the password hashes.)  While the ability to crack such weak mechanisms is old news (WinXP’s LM hashes can be cracked using a rainbow table on one PC, in seconds), consider how easily the five server cluster could become a 5,000 server cluster and how that would work against SHA-1.

Centralized User And Password Databases: Single Sign-On

Rather than a password data file per host, it makes sense to use a single central repository for user authentication information.  This supports single sign-on.  Some choices are: LDAP NIS, NIS+, and Kerberos.  Take care to encrypt passwords while in transit across the network!

All real security policies require detection of old or rogue user accounts, as well as enforcing password strength (see below).  Only SOHOs use per-host password, group, and shadow files.  Larger organizations use a central repository.  In the past NIS/NIS+ were used, but today LDAP and Kerberos are common.  Very large organizations may use PeopleSoft or SAP HR (human resources) systems to manage employees.  You need some way to “diff” the HR system and the list of user accounts.  Note some user accounts for non-employees are always present, e.g., system accounts.

Once you have a list of valid accounts, you must monitor all systems (per host files and LDAP) for any changes.  Nagios, Samhain, or other monitoring systems can do that.  Auditors will want to know how you maintain the legitimate accounts list, and what your incident response policy is when unexpected changes are made.

How Often To Change Passwords

It seems intuitive that to defeat a brute force attack you only need to change your password (“password rotation”) every so often.  A three month lifetime is commonly used in business organizations.  (I believe the three month figure commonly used in business is because DES could be cracked in about that time frame, when this “best practice” was promulgated.)  FIPS-112 required the maximum lifetime of a password is one (1) year.  However, real-world results, as well as mathematical analysis, don’t support this theory.  (Reference: Howard, Mike (mike@clove.com), how often should you change your password? published in ;login: Usenix vol. 31, no. 6 (12/2006), pages 48–51.)

Making users invent new passwords for each account several times a year forces users to pick poor passwords, some variation of “July2010” or worse, or  use the same passwords everywhere (so if any one system is vulnerable, all your systems are).  This is clearly bad security as well as highly annoying to users!  If the same password is used for POP3, IMAP, and other services which transmits usernames and passwords in plain text, it won’t be long before attackers get a copy of those passwords!

It is safer to enforce good passwords and use a strong hashing algorithm (MD5 is okay for medium or low risk situations, SHA–384 or SHA–512 is currently good for high risk situations).  Having very strong passwords and using a strong hashing algorithm should mean you don’t have to change passwords more than once every 3 to 5 years (or never).  This will be safer than using poor passwords that are quickly guessed.

Also consider using smart cards or biometrics (e.g., fingerprint scanner) instead of (or better, in addition to) passwords to authenticate users.

Password Auditing

It is possible to bypass the PAM mechanism and set a weak password (root can do that, or can copy some password hash and paste it in /etc/shadow.)  A good SA will periodically try to crack passwords, just to make sure only good passwords are used.  One such tool is John the Ripper which is easy to install and use: yum install john
unshadow /etc/passwd /etc/shadow >~/pwlist; john pwlist

(As of v1.7.6, JtR doesn’t know about SHA hashes; but there is a patch.  You can DL the source, apply the patch, and build a working version.  Or you can try the new “--format=crypt” option, which should work but is slower.  The current versions do know about modern hash methods.)  Additional checkers exist to audit just Windows passwords, efficiently (useful when using Samba for example).  Other popular tools include the very fast hashcat (uses CPU) and oclhashcat-plus (uses GPU).

Password Cracking

An attacker can defeat password authentication in a number of ways.  (A good reason to use two factor authentication!)

If weak physical security is used, an attacker may be able to see a password as it is typed in.  Also, some people write down passwords.  While not an unsafe practice per se, leaving the paper with the password lying around is.  (As is storing passwords in unprotected files or in emails.)

Passwords can also be stolen with key loggers, a type of malware.

If weak passwords are allowed, many people will chose easily guessable passwords.  An attacker can do a bit `of Internet search for a person to find their name, family members’ names, pet names, cars you’ve owned, the addresses where you or your family has ever lived, etc.  This will yield a few hundred proper nouns and numbers.  By trying combinations and simple variations, a surprisingly high percentage of passwords can be cracked.  In a dictionary attack, a standard English dictionary of words and proper nouns are tried, including simple variations (e.g., reverse, mix in a few digits, strange capitalization, substitute one for ell, zero or oh, etc.)  Using pwquality or other system settings to prevent weak passwords is very effective here.

The secure hashing scheme used (e.g., MD5) may have flaws unknown to any but an attacker.  This is extremely unlikely!  (As mentioned previously, flaws in MD5 were discovered in 2008.  For this reason, Fedora since version 11 uses SHA-256 or SHA-512 everywhere (passwords, package digests, etc.)

It can be effective to limit the number of network attempts per minute to your SSH port.  (Changing that from port 22 will only stop script-kiddies who don’t bother to run a port scan first.  But that may be a lot of attempts!)  You can do this at the network level with iptables; there are several ways it can be done.  Here’s what I used for YborStudent (long lines wrapped for readability):

# Allow unlimited SSH connections from HCC
# (whitelist, so students don't lock out everyone):
-A INPUT -s 169.139.223.1 -m state --state NEW \
   -m tcp -p tcp --dport 22 -j ACCEPT
# Here we limit users to 5 ssh connects per minute,
# or the connection is dropped:
-A INPUT -m state --state NEW -p tcp --dport 22 \
   -m limit --limit 5/min -j ACCEPT
-A INPUT -i eth0 -p tcp --dport 22 -j DROP

The problem with this is low and slow attacks: attacks meant to be slow enough that IDS systems aren’t tripped, but fast enough where the attacker has a decent chance of getting access.  Typically, there are about one or two attempts per minute against the root (or other) account, each coming from a different IP address (maybe a botnet).  While this doesn’t sound like much, it’s enough for an attacker to cycle through 1440 common passwords in a single day.  Admins who use passwords like “root”, “test”, “admin”, “password”, etc., are likely to fall victim.  A good IDS/IPS system such as ossec should be used and tuned to prevent such attacks (and don’t forget using pwquality to enforce strong passwords!) 

The final method is a brute force attack.  Here the attacker tries all possible passwords until one works.  The strength of various encryption and secure hashing methods is measured by counting how hard such a brute force attack would be.  Statistically speaking, if there are N possible passwords, an attacker would on average need to try N/2 of them.  Since only so many passwords can be hashed per second and compared with some hashed result on modern (and predicted) hardware, you can estimate how many days/months/years it would take to crack a password.  The original Unix crypt algorithm would take less than a week on a modern COTS computer.

Today an attacker has additional tools available to speed up the cracking process:

·       A cluster (or bot-net) may be employed to have hundreds or thousands of CPUs working on cracking a single password.  Governments could afford to build special purpose clusters with millions of nodes for this.

·       A special “future” type of hardware that uses quantum computing will be able to search through millions of passwords simultaneously.  So the brute force attack that took years on traditional hardware may only take hours on a quantum computer.

·       Today’s massive disk arrays mean an attacker can pre-compute and store on disk all the MD5 (say) hashes (2^32) for all possible passwords less than a given length  (a rainbow table).  Then, a simple lookup of the hash yields the password.  This is the main reason why modern systems use shadow password systems, where not even the hash is world-readable.

·       The GPU (graphics processing unit, found on high end video cards) can be used as a math coprocessor for computationally intensive, data-parallel computing problems.  NVIDIA and AMD/ATI have been marketing this feature of their cards.  This can be used to dramatically speed up the password-cracking process.

Other Password Issues and Mechanisms

Crypt, MD5, and SHA-1 through SHA-256 are all fast on current hardware.  To make your password hashes more secure you can stretch them by running thousands of hash operations, each on the result of the previous one.  (PHK MD5 does that.)  You start by hashing the password, adding the salt/nonce, and hashing that.  Keep hashing the result a random number of times.  To check a password, you must run through the same number of iterations (unless someone discovers a mathematical trick to avoid doing that; unlikely.)  Or you can use a slower block encryption scheme to generate the hash, such as Blowfish.  (Bcrypt does this.)

Note that besides correctly hashing a password, there are issues about storing it, using a hashed value as a web browser cookie, transmitting the password across a network, time-stamping, generating strong salts, password updating, and other issues.

The bottom line is you should never roll-your-own password or crypto system.  Use a standard tried-and-true solution or at least use standard crypto libraries.

SRP is the Stanford Secure Remote Password protocol.  It is a public key cryptosystem designed to securely store and validate passwords without storing them in the clear or transmitting them in the clear.  This is similar to MS-CHAPv2.  While the University of Stanford holds a patent on this method, it is available for use for free.  The WoW and Warcraft III systems use SRP.  The IETF has standards for it, RFC-2945 (SRP) and RFC-5054 (TLS-SRP).

Never send any passwords through email.  It goes through too many hops unencrypted, gets indexed by search features on email servers, etc.  Login names can be sent this way, but passwords must go by SMS, or voice, or telegram, or postal mail.

Keeping passwords different for every website can be difficult, but there are several techniques you can use.  One is to use a mnemonic technique, combining the site or project name with a standard password you always use (and can remember).

For instance, say your standard password is CatDogRat.  (I would hope it is stronger, something like “:xT9/qwB”; try apg to make one.)  The last few letters (or last+first) of the site name goes between the Cat and Dog, and the number of letters of the site name (or the difference between that and the number of letters in the username used on that site) goes between Dog and Rat.  So my password for wpollock.com would be CatckDog8Rat.  Of course, the standard password should not be words but a normal, strong password.  Using such a mnemonic means I can use a different and strong password for every site, while only having to remember one good password.

Another technique is to use a password manager.  This is software that stores your user IDs and passwords for various websites, mail services, etc., in a securely encrypted file.  To use any of the passwords you only need to remember one (very strong!) master password.  Some popular password manager software is included with web browsers (e.g., Firefox Sync and Xmarks) and MUAs, and there are stand-alone ones such as gpg-agent, ssh-agent, and keyring for Linux and Unix systems.  Some others include LastPass, KeePass, and 1Password.  The encrypted password files can be backed up on a service such as DropBox or Box.net.  With this method, you can use a password generator such as “apg”, “pwgen”, or “pwqgen” to create very strong, different passwords for every site (and for the one master password you must remember).

Microsoft’s Safety & Security Center provides a guide on creating strong passwords, that includes using Microsoft’s online password strength checking service.  See also Google’s guide to creating sensible and more secure passwords.

One last point:  Many websites allow a password reset using just your email address.  So if an attacker gains access to your email, they can reset your passwords!  Protect your email accounts with very strong passwords.

 

Locking Down (System) Accounts

Unix and Linux don’t (and never have!) provide a way to lock accounts.  This is because early on, it was thought that you could prevent logins by manipulating the password hash such that no password could match it.  However, this assumption is no longer true, if it ever was: fingerprint readers and SSH keys don’t use the hashed password field in any way.  Not only that, but many jobs can be run from an account locked in this way, such as cron and at jobs, or jobs started vial the email forwarding system (~/.forward or the equivalent allows email to be piped through an arbitrary program).

Where possible, replace the shell with “/sbin/nologin” (which politely refuses login) or “/bin/false” (which just fails).  This will disable at, cron, and su on many systems.  Note a missing shell (empty shell field in /etc/passwd) implies /bin/sh; however, PAM or other mechanisms may consider a missing shell to lock the account.  Run pwck and grpck, and fix any reported problems.  (Missing home directories are common; Consider using /var/empty or /var/empty/username as the home directory.)

If the encrypted password field of the /etc/passwd file is set to an asterisk, the user will be unable to login using login, but may still login using rlogin(1), run existing processes and initiate new ones through rsh, cron, at, mail filters, etc.  If set to “*NP*” the password is obtained via NIS+.  If an “x”, the /etc/shadow file is used.  If empty, the user can log in without a password (some systems lock the account if the password field is empty).  Otherwise it contains the password hash.  (On older Unix systems, this can be followed by a comma and then password aging fields separated with commas.)

Enabling/disabling (unlocking/locking) accounts is done in a system-dependent way.  Mostly commonly, accounts are locked by changing the password hash in /etc/shadow.  On some versions of Unix, this may also prevent creation of new processes via at, cron, etc.  In shadow, an account is locked if the password field is (or starts with) ‘*LK*’ on Solaris and UnixWare, ‘*’ on HP-UX, containing ‘Nologin’ on Tru64, a leading ‘*LOCKED*’ on FreeBSD and a leading ‘!’ on Linux.  (Empty or null passwords are locked with “!!” instead.)  At least on Linux, using “*” or “no login” in the passwd file’s password field also locks an account.

Modifying the password hash makes it impossible for any entered password to match.  If the password field is prepended and not completely replaced, an unlock operation can restore the previous password.  On other systems, the tools locking passwords replace the hash completely, so a new password must be set to enable the account.

Generally, you lock and unlock accounts with passwd -l (and “-u”) or usermod -L (and “-U”), rather than edit account files by hand (say by using the vipw command).

Locking an account by changing the shell field yields the same result, and additionally allows the use of su.  The system maintains a list of valid shells (often in /etc/shells).  Any account with an invalid shell (any program not in the list) is considered locked (by pam_shells).  Using an invalid shell (often /bin/false) may prevent other access as well (e.g., FTP; see pam_shells).  Unfortunately, an account with an invalid shell is still a valid account as far as PAM (pam_unix) is concerned!

 Using an invalid shell may prevent su and may prevent cron jobs and at jobs as well (depends on your system).  Locking an account may prevent some services for the user.  For example, on some systems sendmail won’t deliver email to a locked account.

Remember that locking a password doesn’t prevent login or remote command execution via ssh (or scp or sftp)!  (User can still use SSH keys, which bypasses PAM’s auth modules even when sshd is configured to use PAM.)  You should also change the shell to an invalid one.

Some versions of sshd do check for locked accounts, when not configured to use PAM.  When configured to use PAM and sshd uses keys to authenticate the user, then it only uses the PAM account modules, not the auth modules, and pam_unix doesn’t check for locked accounts.  (The designers apparently assumed the authentication will fail if the account is locked.)  While Linux PAM doesn’t contain any module that can check for locked accounts in the account context, you may be able to find one you can use from a list of non-standard PAM modules on the Internet, or create one, but make sure the module you find can work in the account context and not just auth.

The standard Linux OpenSSH server sshd supports an internal list of allowed (or denied) users, but you must update that whenever you lock or unlock any account.  There are PAM modules such as pam_listfile that also use such lists, but they only work in the auth context.  (Note that not all *nix systems have this bug.)

Most sshd servers can be configured to use PAM, and a PAM module could check for locked accounts (currently there isn’t one).  Also, sshd can deny access to a configured list of users or groups.  (See sshd_config).

Summary: For maximum safety, lock accounts by both setting an invalid, non-working shell and locking the password, after updating PAM to check for valid shells in both the auth and the account contexts.  Don’t rely on just passwd -l to lock passwords.  Configure SSH to use PAM.  Consider creating a PAM module to check for locked accounts, if your system doesn’t provide one.

Database Security

A database (DB) is simply a collection of related data.  Simple files are often used as databases, such as /etc/passwd.  More complex situations use a Database Management System (or DBMS) to gain extra performance, support multiple users, allow remote access, etc.  Common DBMSs include Oracle, PostgreSQL, MySQL, Jet (MS Access), SQLserver, DB2, and others.  Each can manage multiple independent DBs.

Ultimately a DMBS stores data in files.  So securing a DB means first securing its files by choosing an appropriate filesystem, using RAID hardware, setting correct mount options, performing regular backups, using monitoring tools, and of course setting the proper file permissions.

Beyond this, a DBMS has its own security system to limit access and to provide audit logs.  For each DB a DBMS allows you to specify a set of users and passwords (or other authentication tokens), and for each what data the user can access, modify, or add.  Normally this list of users and passwords is independent of the system users and passwords.

Depending on the DBMS used, access may be limited to the tables, the columns, or the rows in a DB.  In addition, a view can be defined.  A view can be considered as a virtual table, defined in terms of other tables, and is handy to limit what a user can access (i.e., instead of defining tedious rules of which parts of which tables can be accessed, you create a view that only shows the accessible data, and allow users access to the view rather than the underlying tables).  Access to such database objects is controlled with the grant and revoke statements.

With PostgreSQL, access is separated from authentication.  User authentication is controlled by the file pg_hba.conf.  The usual default is to authenticate all local (system) users.  See the pg_hba.conf file documentation for details.  Users can be defined with the create user command.

With MariaDB (MySQL) and other RDBMSes, user authentication is done with nonstandard grant syntax.

For remote users (the common case), you should enable TLS security for the connections.

You should consider database (or whole disk) encryption.  This might be required depending on the type of data stored (financial, health records, etc.).

Finally, be sure access to and performance of your database is monitored.  You need to watch for unusual activity, such as very large transactions, number of connections, access denied log messages, very slow response times, many password attempts, etc.

Preserving Integrity with Normalization

Another problem is preserving the integrity of a DB. There is little that can be done to prevent external inconsistencies, often the result of operator input errors.  A poorly designed DB can suffer from internal inconsistency (also very poor performance).  Internal inconsistency occurs when updates to part of the DB conflict with other parts.

The solution is to use normalization, a process of transforming a poor DB design (called the schema) into a good design, one where internal consistency is preserved.  There are many steps to normalization, but it is common to use only the first three (which is almost always good enough).  Each step is called first normal form, second normal form, and third normal form.

Discuss referential integrity and foreign keys.

Review AUnix1/DatabaseBasics.htm.

Demo: Create DB and secure PostgreSQL DB.  Show user management, views, referential integrity, not NULL.

Project: design a properly normalized DB.  Allow three users: admin (total access), user1 (write access to one table, read for rest), user2 (read-only to some views).  Make sure DB uses referential integrity.  Implement in PostgreSQL and MySQL.  Consider monitoring and backup.

Other Datasbase Security Issues

Distributed databases and “big data” applications have some interesting security issues.  Multiple nodes must agree on changes made, even with the threat of external actors and malicious or erroneous nodes in the network, and even if some parts of the network are unreachable.  Data integrity and accountability must be possible.  The issue is called the consensus problem, and is addressed by using special protocols such as paxos and/or blockchain technology.

Lecture 9 — File Permissions

*nix systems (including Mac OS X and other POSIX compliant systems) have a simple system for controlling access to files and directories.  And since other resources such as disks, ports, etc. have file names (under /dev) you can also control access to these devices.  Windows filesystems up through Windows ME don’t support permissions.  (Just a “read only” attribute.)

This system works by assigning a user (UID) and a group (GID) to every file.  Then, users (processes’ owners) of a file are put into one of three classes: the owner (the process’ UID matches the user of the file), the group (not the owner, but the process’ UID is a member of the file’s group), and other (everyone else).

For each class of user, there are three possible permissions that can be granted: read, write, and execute.  Any attempt to access the file’s data requires read permission.  Any attempt to modify the file’s data requires write permission.  Any attempt to execute the file (a program or a script) requires execute permission.

In *nix systems, directories are also files and use the same permission system as for regular files.  Note permissions assigned to a directory are not inherited by the files within that directory.

Because directories are not used in the same way as regular files, the permissions work slightly (but only slightly) differently:

·       An attempt to list the files in a directory requires read permission for the directory (but not on the files within).

·       An attempt to add a file to a directory, delete a file from a directory, or to rename a file, all require write permission for the directory (but, surprisingly, not on the files within).

·       Execute permission doesn’t apply to directories (a directory can’t also be a program).  But that permission bit is reused for directories for other purposes.

Execute permission is needed on a directory to cd into it (that is, to make some directory your current working directory).  You need execute permission on a directory to access the inode information of the files within.  You need this to search a directory to read the inodes of the files within.  For this reason the execute permission on a directory is often called search permission instead.

Search permission is required in many common situations.  Consider the command “cat /home/user/foo”.  This command clearly requires read permission for the file foo, but unless you have search permission on /, /home, and /home/user directories, cat can’t locate the inode of foo, and thus can’t read it!  You need search permission on every ancestor directory to access the inode of any file (or directory), and you can’t read a file unless you can get to its inode.

Various other commands will need to access the inode of files to work.  Earlier I said you need write permission on a directory to add, rename, or delete files within.  But all those actions require changing or at least reading the inodes of the files within, so search permission is also needed.

Most systems also support some kind of access control lists, either proprietary (old HP-UX ACLs, for example), or POSIX.1e ACLs (a POSIX draft that was abandoned but is still used) or NFSv4 ACLs, which are the part of NFSv4 standard.  These are discussed below.

Review inodes (three time stamps: atime, mtime, ctime, also called “MAC times”.  Ext3 adds a “dtime” which is when the file was deleted).  Review other inode information.  See /usr/include/ext2fs/ext2_fs.h for a list of reserved inodes (1-10, #2=root) for ext* filesystems.  The atime is updated when the file’s data is read.  The mtime is updated when the file’s data is changed.  The ctime is updated when the file’s inode data changes.  Note that changing the mtime, permissions, or anything except atime causes ctime to be updated as well.

Other filesystem types have different kinds of inodes (and may not call them that).  The OS maps the actual filesystem’s data to standard inode data (e.g., FAT, NTFS).  Defaults are provided for missing permission data (e.g., FAT), while extra permission data is ignored (e.g. NTFS).  So all files appear to have standard inodes.

Review links (hard and soft).

Discuss FilePermisions.htm doc, especially SGID on a directory (e.g., a web author’s group).

EUIDs

While every user has a UID assigned, every process has several:  The effective UID (EUID) is the UID used to determine file permissions.  When running a SUID program (any program with the setUID bit turned on), the EUID changes; the real UID (RUID) does not.  For example, running a SUID instance of the dash shell and the id command shows:

  uid=500(wpollock) gid=500(wpollock) euid=3(adm) ...

UIDs (and similarly for GIDs) can be specified as real or effective.  Assigning effective IDs is preferred over assigning real IDs.  Effective IDs are equivalent to the setuid feature in the file permission bits.  Effective IDs also identify the UID for auditing.

However, because some shell scripts and programs require a real UID of root, real UIDs can be set as well.  For example, the reboot command requires a real rather than an effective UID.  If an effective ID is not sufficient to run a command, setting the SUID bit may not be sufficient.  You need to assign the real UID to the command as well in such cases, using a wrapper program that in turn runs the program.

The same applies to the GID as well.

My wrapper.c is very insecure (and doesn’t work if the real UID is root): the C system function is trivially hacked, say by creating your own version of bash and changing PATH to use it; or by redefining IFS to include “/” and creating a program named bin that does evil things, or simply by mucking up the environment (PATH, umask, ...).  A better wrapper.c uses execv (?) not system, uses absolute pathnames, and “scrubs” the environment (deleting the environment including aliases and functions, then resetting PATH, umask, HOME, etc. to correct default values.  (Left as an exercise to the reader.)

Show and discuss SUID Demos (reader, wrapper.c on YborStudent).

Mention the wheel and sysadmin groups (members have rootly powers on some systems).  (On Fedora, only members of group wheel can use sudo and perform other admin tasks that use polkit (similar to sudo, but a DLL used mainly with GUI applications).  Such users are “Administrator” as opposed to “standard” users, when managing uses from the GUI.)

Discuss using setGID (and possibly the sticky bit) to create a workgroup directory.  (FreeBSD also allows setUID on directories; requires a special mount option.)

All world-writable directories should include the sticky (text) bit: /tmp, /var/tmp, and /dev/shm.

Additionally, in Linux 3.6 and newer, a hard link to a file can only be created if the user owns the file or has write access to it.  This prevents the common trick used by attackers to escalate their privileges by using background services running as root.  See /proc/sys/fs/protected_hardlinks in proc(5) for details.  Modern Fedora enables this protection by default, in /usr/lib/sysctl.d/50-default.conf.  To override that, copy that file into /etc/sysctl.d/, and edit that copy.

chmod [ugoa]{+-=}[rwxXstugo]:  +X means to add x if dir or has x for someone; g=u means to copy owner perms to group.

Files attributes:

In addition to the standard permissions, some filesystems support file attributes or flags.  (BSD’s UFS does, Solaris’ UFS doesn’t.  Linux ext* filesystems do.)  For Linux, use the commands: lsattr, chattr:  chattr +|-attributes
The ext* attributes include:  A=don’t update atime, S=synchronous updates, D=synchronous directory updates, a=append only, c=compressed, d=no dump, i=immutable, j=data journaling (not useful for ext[34]), s=secure deletion, T=top of directory hierarchy, t=no tail-merging, and u=undeletable.  See the man pages for more details.  (Attackers often set these, especially “i” on infected files.)

For security (integrity), consider setting the S and D flags on database files.

Since DB files are very large and constantly changing, backups of them are wasteful; unless the DB was quiesced, the backups will be corrupted anyway.  Normally, you backup databases with special tools.  If that is the case, you could add the d flag (no dump) as well.

The immutable bit (i) is useful to prevent some attacks and some errors.  Most script-kiddie attacks won’t account for that, and their failure should show in your logs.  (This bit prevents files from being opened for writting, but if the file is already open, does not prevent modification.)  Critical config files can be made immutable as well, such as /etc/ssh/*.  Finally, some malware creates files to compromise your system.  You can prevent that by creating empty, immutable files.  For example, ~root/.rhosts (if you are foolish enough to enable rsh), dhclient.conf, and similar files.  Note the i bit is used on aquota.* files by quotaon, presumably to prevent you from running quotacheck without turning them off first.  (If your system crashes, you may need to unset that manually or quotacheck may fail!)

You should consider which attributes (when available) are more useful than problem-causing and enable them.  One problem-causing attribute is the a flag.  It seems obvious to use that on log files, but doing so may prevent log rotation!

File Commands Review, Files with Spaces in Names

Review mv, cp, rm, ln, touch commands, and umask, chmod, chown, chgrp, commands.  su (for “switch user”, not “super-user”), su – (Runs root’s login scripts), su [-] user.  Note su [-] [user] command, the command is passed to sh (for example, “su -c 'something'”).

Show security hole with find and xargs: since file names can contain any characters except for “/” and NUL, the following creates a file that appears as two files to the rm command: cd /tmp; mkdir -p 'foo\n\etc'; touch foo 'foo\n/etc/passwd' then: find /tmp ... |xargs rm -f

(The fix is to use -print0 with find and -0 with xargs.  These options separate filenames with NUL and not newline.)

ACLs (or Access Control Lists)

Using groups to define permissions can be tedious.  You need to define one group for each set of users with the same permission requirements per set of files.  Changing the groups of such files, and adding and removing users to each of the dozens of groups, can be time-consuming, tedious, and error-prone.  New files may need their group changed, a step often forgotten (but SGID on a directory helps.)  Restoring files from archives may cause unexpected access to the files as the group definitions change over time.

Using ACLs may be a much better approach if available.  Both the OS and the filesystem must support these.  Common to Linux and Solaris is POSIX ACLs.  The idea is that a given file (file system object) can have multiple users and groups associated with it, each with its own set of (three) permissions.  It is no longer necessary to define new groups.  In addition, directories can be given default ACLs that will be added to any new files added to that directory.

POSIX ACLs (a.k.a. draft POSIX ACLs)

Many systems (including Linux) support POSIX ACLs (as of 2009).  Both the FS and kernel must support ACLs.  The FS must be mounted with -o acl.  If a file has ACL entries the “ls -l” output includes a “+” after the permissions.

The POSIX ACL commands affect all users and groups with permissions on a file, except for the owner (and “others”).  You can view and set/change/remove POSIX ACL entries (on systems that support them) with:

getfacl file
setfacl [--test] -m acl file..., acl = {u|g|o|m}:[name]:[rwxX]+, where ‘m’ is mask, ‘X’ (cap X) means execute only if directory, or if some (other) user already has execute.  Use “-m acl” to modify, “-s acl” to replace (set) with new, and “-x acl” to remove.  Use “-b” to remove all, and “-k” to remove default ACLs.  Use “-R” to apply ACLs recursively.

Because ACLs can be complex it was realized that it is easy to create files with more access than intended.  To control this, an effective rights mask is also defined.  This mask defines the max permissions allowed for any ACL entry as well as the owning group, but not the user owner or (oddly) “others”.

As a “convenience” when changing ACLs, the mask is recalculated to the union of all assigned ACLs (including the owning group).  That will undo any explicit setting of the mask.  Use the “-n” option to prevent this.

Directories can be assigned an additional set of ACL entries, the default ACLs.  Files created in a directory that has default ACL entries will have the same ACL entries as the directory’s default.

Use “-d” to affect the default ACL (useful on directories), or use an ACL prefix of “d:” (ex: d:m::rX or d:u::rx).  Examples:

setfacl -m u:ud00:r file    # add read for ud00,
setfacl -m d:u:ud00:r dir # set default ACL to read for ud00,
setfacl -x u:ud00 file         # remove ACL entry for ud00,
setfacl -m m::rx                       # remove write permission from all,
setfacl -b file           # remove all ACL entries from file,
u::rw-,u:ud00:rw-,g::r--,g:staff:rw- # more complex example.

See acl(5).  Remember: FS, kernel must support; FS mounted with -o acl.  (This is the default for ext4 as of 2012; use the noacl option to disable.)

Most FSs use extended attributes to hold ACLs (not related to ext* attributes).  A pointer in the inode points to a single block holding these.  Thus the total size of all attributes (SELinux labels, ACLs, and any others) must be less than 1 KiB.  (See {get,set}fattr(1), attr(5): Show: getfattr -dm - path
(There is a mount option for this as well, for some FSes.  Like the acl option, it is enabled by default on modern Linux.)  Note attribute names that don’t start with “user.” are restricted to root and/or system use only.  You can define your own attributes to use with your own tools and scripts (e.g., last-backup-date, type, icon-data, ...).  (Ex:  setfattr -n user.foo -v bar file)

Files created in a directory that has default ACL entries will have the same ACL entries as the directory.  New sub-directories will get the default ACL entries too.

Many archiving tools ignore extended attributes (and hence, ACLs).  Backup and restore (per filesystem) by saving ACLs to a file, then applying the file.

Backup: cd /home; getfacl -R --skip-base . > ./backup.acls
Restore: cd /home; setfacl --restore=./backup.acl

Or use an archiving tool that supports extended attributes, ext* attributes, and ACLs: star H=exustar -acl -c path >archive.tar
and to restore use: star -acl -x <archive.tar

(The POSIX standard tool pax can support this, but only when using the non-default archive type “pax”.  Run “pax -x help” to see if the pax archive type is available on your system.)

Example 1:  A project directory (e.g., the website) can have permissions of drwxrwsr-x   root   www, which means any new files added will also have group www.  (Setting the sticky bit can be good too, to prevent anyone except the owner from deleting the file.)  However, with the default umask that is often used, the new file’s permissions will be -rwxr--r-- and this won’t allow other group www member to edit the file.  The solution is to set a default ACL on the directory as well, effectively overriding umask:
    setfacl -R --set d:g::rwX /var/www/html

Example 2:  Older rsync versions have no options to set/change the permissions when copying new files from Windows; umask applies.  So when uploading a website, no files are readable!  One way to fix this is to run a cron job regularly to fix permissions.  A better way is to set a default ACL on each directory in your website.  Then all uploaded files will have the umask over-ridden:

    cd ~/public_html # or wherever your web site is.
    find . -type d | xargs setfacl -m d:o:rX

This says to set a default ACL on all directories, to provide “others” read, plus execute if a directory.  (New directories get this default ACL too.)  With this ACL, uploading a file will have 644 or 755 permissions, rather than 640 or 750.

NFSv4 and ZFS ACLs [From: opensolaris.org/os/community/zfs/docs/zfsadmin.pdf]

Defined in the NFSv4 specification, a new ACL model (sometimes called NT-style or NFSv4-style ACLs, or just NT-ACLs) fully supports the interoperability that NFSv4 offers between UNIX and non-UNIX (i.e., NTFS and SMB) clients.  (NFSv4 ACLs and Windows ACLs are the same except for the format of the username.)

NT-ACLs provide much richer semantics than old-school POSIX permissions (and ACLs).  ZFS and NFSv4 support this new ACL model and no longer support the POSIX-draft model.  (ZFS “fakes” standard permissions for the “ls -l” output.)

NT-ACLs use chmod to change, and ls -v to view.

In the new model files and directories can have multiple access control entries (or ACEs).  Unlike POSIX ACLs there are two types of ACE: ALLOW and DENY.  These are examined, in order, to determine if access is ultimately allowed or denied.  Only ACEs that have a who that applies to the current user are examined.

If some ACE grants access, a subsequent deny ACE has no effect.  So always put the deny ACEs first, then the allow ACEs last.

NT-ACLs provide 17 different permissions, unlike POSIX ACLs that only provide read, write, and execute/search.  You can set the new ACLs for a file’s owner, group, or others using a special syntax: owner@, group@, and everyone@.  Beyond that you can set ACLS for any user or group with user:name or group:name.

NT-ACLs have several different inheritance modes (besides none).  (See the table below.)

Each ACE is roughly three parts: who it applies to, which set of permissions, and if the permissions are allowed or denied.  This syntax allows the SA to (for example) add an ACL that denies all members of some group the permission to add a file in some directory, and also make exceptions for specific users by adding additional ACEs to allow the permission.

The Solaris syntax to prepend (not append) a new ACE to the ACL list is:

chmod A+who:permission[/permission]...[:inheritance-flags]:{allow|deny} file...

You can replace an ACE with “Aindex=...”, remove one with “Aindex-...”, insert a new ACE after an existing one with “Aindex+...”, and remove all extra ACEs with “A-”.  The permissions and inheritance-flags can be specified with a verbose or a compact notation, for example “r” or “read_data”.

Here’s an example:

# chmod A+user:gozer:read_data/execute:allow test.dir

# ls -dv test.dir

drwxr-xr-x+ 2 root root 2 Feb 16 11:12 test.dir

0:user:gozer:list_directory/read_data/execute:allow

1:owner@::deny

2:owner@:list_directory/read_data/add_file/write_data/add_subdirectory

/append_data/write_xattr/execute/write_attributes/write_acl

/write_owner:allow

3:group@:add_file/write_data/add_subdirectory/append_data:deny

4:group@:list_directory/read_data/execute:allow

5:everyone@:add_file/write_data/add_subdirectory/append_data/write_xattr

/write_attributes/write_acl/write_owner:deny

6:everyone@:list_directory/read_data/read_xattr/execute/read_attributes

/read_acl/synchronize:allow

ZFS has been ported to many systems, and work on supporting NFSv4 ACLs on other *nix systems is progressing, so it is probably only a matter of time before an SA will need to understand the new ACLs.  See the ZFS Admin Guide and the NFSv4 docs (RFC-3530) for more details.  Also all (or at least Linux) systems that support NFSv4 support the new ACLs.  (Fedora 8 and newer does; yum install nfs4-acl-tools.)

(Note that Solaris 11 permits chmod to add either POSIX or NT ACLs!)

File Locking  [Expanded from Admin 2, Shell Scripting.  See also flock-demo.]

Some administrative tasks that involve updating config files (or any shared data files) can be dangerous.  A conflict can arise if two or more processes attempt to modify the same file at the same time.  For example, consider what might happen if you edit the /etc/passwd file while someone else (or a dnf update running from cron) updates the file (or a related one such as opasswd, gpasswd, shadow, ptmp, ...) at the same time.  Or one process modifies shadow while another modifies passwd; the files are no longer consistent and may even become corrupted.

Another common problem is with daemons.  Starting one while a previous one is still running can cause the running process to fail.

File locking is used to prevent this sort of error.  Before opening some file, a process obtains a lock on it.  While locked, no other process can open the file and hence must wait until the first one is done before having a chance to obtain a lock of their own.

A system admin must sometimes manage, troubleshoot, or setup locking.  This may involve mount options, setting special permissions, creating/removing lock files, and even tracing utilities to see what locking they use, if any.  Accordingly, you need to know something about them.

There are different types of locks used:

·       Locks can be shared (read-only) locks that prevent anyone from writing to the file while allowing read-only access, or exclusive locks for writing (and security) that prevents any other process from accessing the file while locked.

·       If the kernel enforces the locks they are called mandatory while if not they are called advisory (only supported by a shared library perhaps, so apps not using the library aren’t forced to obey the locks).  Not even root can violate a mandatory lock!

·       Locks that work on a range of bytes within a file are called record locks; otherwise whole files are locked.

·       Locks that force a process to wait to acquire it are called blocking; the process is frozen until it can obtain the lock.  Locks that simply return failure immediately (when trying to acquire one held by another process) are non-blocking.

Different *nix systems support different locking schemes.  See flock(2), fcntl(2), and lockf(3).  Flock was introduced in BSD.  The original implementation had problems and later versions work in incompatible ways.  POSIX defines both historical fcntl and newer lockf locking, but not flock even though it is arguably better.  Note that flock(1) can be used from the shell.

On most systems, all three locking mechanisms are supported for advisory locking, but are independent of each other.  (Nasty when you don’t know which scheme was used by other utilities!)

Qu: how can you determine which lock mechanism was used by some utility?  Ans: strace:

  strace -fFo pw.out passwd; grep -E 'fcntl|lockf' pw.out

(Another way: run the lslocks command.)

Only POSIX fcntl locks are honored by NFS.

On Linux 2.6, lockf is implemented as fcntl (so you have one less scheme to worry about.)

Sys V Unixes (including Solaris, HP-UX, and others) and Linux support mandatory locks, even though POSIX only requires advisory locks.  In the Sys V scheme, a file could use mandatory locks by setting the SGID bit on and the group execute bit off.  (This is otherwise a meaningless combination of permissions.)  (Solaris also allows chmod +l; ls -l will show either the l or the s in the group field.)

Linux permited Sys V mandatory locking on filesystems that support it (including ext*), if that FS is mounted with the mand mount option and the file has g+s,g-x permissions.  (Show, but don’t demo lcktest.c.)  If so, fcntl locks become mandatory.  flock is still advisory.  (On Linux, remember lockf is implemented by fcntl.)

(Summary: Some filesystems, if mounted with mand, if file has g+s,g-x, has mandatory locks with fcntl locks.  The kernel feature for this is disabled as of Fedora 27 however.)

Open File Description Locks (old name: file-private locks)

In Linux 3.15, a new type of locking mechanism was added.  This felt to be needed, because of two major issues with POSIX locks, and one issue with BSD locks:

1.    POSIX locks are per-process, not per-thread.  When a request for a lock comes in that would conflict with an existing lock that the process set previously, the kernel treats it as a request to modify the existing lock.  Thus, POSIX locks are useless for synchronization between threads within the same process.

2.    All locks held by a process are dropped any time the process closes any file descriptor that corresponds to the locked file, even if those locks were made using a still-open file descriptor.  So if a program opens two different links of a hard-linked file, takes a lock on one file descriptor and then closes the other, that lock is implicitly dropped even though the file descriptor on which the lock was originally acquired remains open.

3.    BSD-style locks (using flock) have better semantics.  Whereas POSIX locks are owned by the process, BSD locks are owned by the open file.  So if a process opens a file twice and tries to set exclusive locks on both, the second one will be denied. Thus, BSD locks are usable as a synchronization mechanism between threads as long as each thread has a separate opened file.  Also, BSD locks are only released when the last reference to the open file on which they were acquired is closed.

However, BSD locks are whole-file locks.  POSIX locks, on the other hand can operate on arbitrary byte ranges within a file. While whole-file locks are useful and common, they are not granular enough for some cases (such as a database locking a record).

The new locking scheme is aware of the older POSIX scheme; it’s basically BSD-style locks with byte-ranges, using POSIX (fcntl) system calls.  See “Open file description locks” in the fcntl(2) man page and this article.

Lock Files

System utilities, user applications, and shell scripts often create locks by creating lock files.  When starting, they check if the lock file exists already, and if so either wait or abort.  If there is no lock file already, they create it.  This scheme depends on all utilities accessing the same resource use the same lock file.  So, always document any lock files used by your admin scripts.

It also depends on checking and creating a lock file atomically, which is often not done correctly in shell scripts.  One common but incorrect method is to attempt to create a symlink; if it already exists, this will fail.  However, the only (POSIXly) correct way is the same as for temporary files; turn on no-clobber mode and attempt to create the file using output redirection:

set -C  # or set -o noclobber
: 2>/dev/null >lock-file
if test $? = 0; then got lock...; rm lock-file; fi

Here’s some code to wait for a lock to be released:

set -C  # or set -o noclobber
while ! { : 2>/dev/null >lock-file; }
do sleep 1; done  # loop until lock file created
# got lock...
rm -f lock-file

Often these lock files are empty, but sometimes they contain data such as a PID.  Most system lock files are kept in /var/lock or /var/run.  You can use the old lslk or more modern lslocks to examine these easily.

When working with /etc/passwd and related shadow suite files, lckpwdf(3) is used.  Rather than lock multiple files, this creates an advisory lock on the single file /etc/.pwd.lock (created if it isn’t already there).  vipw and vigr additionally use /etc/ptmp as a copy of the file being updated; when done this is “mv”-ed to replace the original file.

Normally utilities trap signals to allow them to clean up lock files (and other resources).  But, when some program crashes it may not have a chance to remove its lock files.  In that case, you will have problems running the program later.  Many applications (e.g., Firefox web browser) may create lock files that occasionally you must manually remove.  The fix is simple; just manually remove the offending lock file.  Finding the correct file is the hard part!  Google or strace can be used for this.

Here’s some sample code you could use, that checks for lock files containing PIDs (a common case) before starting a daemon (here, we do a sleep 60):

#!/bin/sh
LockDir="/tmp"  # /var/run or /var/lock is common
PidFile="$LockDir/${0##*/}.pid"

AlreadyExists () {
    read RunningPID < "$PidFile"
    if grep -q "${0##*/}" /proc/$RunningPID/cmdline \
      2>/dev/null
    then  echo "$0 is running as PID $RunningPID"
    else  echo "Stale $PidFile found from PID" \
            "$RunningPID.  Overwriting."
          DoIt
    fi
}

DoIt () {
    echo "$$">"$PidFile"
    echo "Started ${0##*/} with PID=$$"
    sleep 60 # Replace this with real work
    rm -rf "$PidFile"
}

(set -C; echo 2>/dev/null "$$">"$PidFile") \
  && DoIt || AlreadyExists

Filesystem Security

Use mount options to limit untrusted file systems.  Example:  Create a floppy with suid to root version of bash, or world-readable/writable version of /dev/mem or /dev/kmem, then mount it at YborStudent.

Useful security related mount options: nosuid, nodev, noexec, and ro (for forensic examination).  (noatime for flash drives is also useful.)

Hiding Data

Files can be hidden using OS features and bugs.  On *nix systems files can be hidden by mounting a filesystem on top of a non-empty directory.

Files don’t get deleted until the hard link count goes to zero.  Even then, the file won’t be deleted until it is no longer in use by any process.  A technique (no longer used much) is to create hidden files by creating them, opening them, and immediately deleting them.  The directory entry goes away but the file can still be used by the process.

Use shred to make it harder to recover deleted files on older hard disks.  (-v = show progress, -u = remove file when done.)  Newer disks and SSDs don’t benefit from repeatedly writting data over files; over-writing them once with zeros (using dd works well) is sufficient, and won’t shorten the life of SSDs as much.

ATA disks allow sneaky commands to permit the disk to hide some space from the OS, using a Host Protected Area (HPA) (since v3) and Device Configuration Overlay (DCO) (since v6).  Actually, at boot time the OS can access the HPA then lock it down, so Microsoft can hide stuff there.  The HPA and DCO are hard but not impossible to access with special software.

ATA (and newer disks such as SATA) support several security features, normally disabled.  Some BIOSes or EFI firmware may enable these features, and may have settings changable from their user interfaces.  If not “frozen” (locked down), you can examine and change these settings using hdparm.  One useful feature is the enhanced secure erase, which erases every sector on the disk (and resets the security settings as well).

Some stand-alone utilities bypass filesystems and can access the raw disk by sector numbers.  This allows data hiding in un-allocated or un-partitioned sectors, and the slack space between the end of filesystems and the end of the volume (partition).  Additionally there are “reserved” and unused areas in most filesystems that can hold hidden data, e.g., an unused sector used to hold boot loader, when the boot loader is in the MBR instead.

Other utilities allow software to read/write NVRAM (CMOS or BIOS memory), both on the motherboard and also any peripheral device with flash-able memory.  This is another place data can be hidden.

Encrypting Storage Volumes

Most OSes today support storage volume encryption, but a better solution is often to use the open-source TrueCrypt software.  Encrypting a volume this way can provide plausible deniability; you can just claim you wiped the drive and an examiner can’t tell.  (With a bit of work, you can use this to secure flash disks too.)

In 5/2014, the anonymous maintainer of TrueCrypt shut down development in a surprising announcment, indicating the software may have serious flaws.  (It was also reported the developer used TrueCrypt exclusively on Windows XP, and once Microsoft dropped that, he had no further interest in the project.)  TrueCrypt is currently being audited, and if found clean, perhaps someone will take over the project.  Update:  Several expensive and extensive code reviews later have revealed some minor flaws but none that puts encrypted disks at risk.  And although the official web site is gone, digitally signed copies of (the last version of) TrueCrypt are still available for download.

There is no widely used, cross-platform, open source replacement for TrueCrypt.  However, there are plenty of filesystem-specific ones and some cross-platform but proprietary ones.  This include Bitlocker for Windows, File Vault for Apple systems, Geli for FreeBSD, and md-crypt/LUKS for Linux.  See Wikipedia for a comparison of some of these.  Currently (2018), I think LUKS is your best option for Linux.

Update:  VeraCrypt is an open source fork of the last available version of TrueCrypt.  It is actively being maintained, and the developers have (or are planning to) address the flaws found in the audits done on TrueCrypt.  Hopefully, they will add UEFI support; currently, like TrueCrypt you need BIOS, and fewer devices support that each year.

LUKS

LUKS stands for Linux Unified Key Setup.  LUKS enables the system to encrypt a whole storage volume (any block device) in Linux for security purposes.  This is an advantage for people who use mobile devices for their day to day work; if the entire disk is lost, your data cannot be easily accessed as it is encrypted and requires a passphrase or password to access the data inside.

LUKS works by encrypting block devices, including disk partitions and logical volumes.  To use the filesystem (or swap data) within the encrypted block device, the admin must enter the password/passphrase to unlock the volume.  Once unlocked, the volume can be accessed normally through the use of a special driver, dm-crypt.  To access the protected filesystem, you use the /dev/mapper/name for the unlocked device, to use with mount, fsck, tune2fs, backup, etc.

To encrypt or to access an encrypted block device with LUKS, the admin uses the cryptsetup utility.  This program can work with other encryption types too.

Suppose you have a volume group “vg” with a logical volume in it, “lv1”, and you wish to encrypt it.  You would run the command like this:

# cryptsetup luksFormat /dev/mapper/vg-lv1

WARNING!

========

This will overwrite data on /dev/mapper/vg-lv1 irrevocably.

Are you sure? (Type uppercase yes): YES

Enter LUKS passphrase:

Verify passphrase:

Command successful.

You can view data about a LUKS protected device with the command “cryptsetup luksDump /dev/mapper/vg-lv1”.  To use the device, you need to unlock (“open”) it first with:

# cryptsetup open /dev/mapper/vg-lv1 myluks
Enter LUKS passphrase for /dev/mapper/vg-lv1:
key slot 0 unlocked.
Command successful.

That command makes /dev/mapper/myluks refer to the unlocked volume.  After this, the device is open and the dm-crypt driver will keep the key in memory, allowing you to use the device with the name you provided: you can now mount, fsck, tune2fs, or backup the filesystem using /dev/mapper/myluks.  When done, you can run cryptsetup close to remove the unlocked device and to wipe the key for memory.  (Note: LUKS supports having several different keys, so you can make one for emergencies (such as forgetting the regular key) and lock it in a safe.)

To shutdown your system, you need to unmount the open block device first, then remove that with “cryptsetup close /dev/mapper/myluks”.

To resize an LVM encrypted with LUKS, you need to run the cryptsetup resize command after resizing the logical volume, so dm-crypt will notice the new size.

You can have an encrypted volume unlocked and mounted automatically at boot by adding an entry to /etc/crypttab for the device, the open (unlocked) name, the key file, and options; dm-crypt will open (unlock) it at boot time.  Then, the entry to mount the unlocked mapper name should work normally (say, via an entry in fstab).  Otherwise, you are prompted for the password/phrase.  See the crypttab(5) man page for details.  Here’s an example:

# mkdir /myluks # the mount point
# echo '/dev/mapper/myluks /myluks ext4 defaults 0 0' \
   >>/etc/fstab
# umask 077
# echo "myluks /dev/mapper/vg-lv1 /etc/myluks.key \
   luks,nofail" >> /etc/crypttab
# printf 'secret' > /etc/myluks.key

(Note the use of printf; using echo will not work, since that adds a newline to your key file!)  Naturally, don’t use “secret”.  In fact, you can create a good key with the command:

   # dd if=/dev/urandom of=/etc/keyfile bs=1024 count=4

LUKS allows you to have multiple keys, so this 4 KiB random number can be one of the keys, and a secure password or passphrase for you to type can be another.

Automatic unlocking defeats the whole point of encryption on a mobile device, but may be useful for a physically secure server.

Lecture 10 — Intrusion Detection Systems (IDS)

Host-based intrusion detection systems (HIDS), as the name implies, are installed on each host and look for attacks directed at the host.  (That is, they don’t examine network data that doesn’t go to the host, and may not examine network data at all.)  Most HIDS employ automated checks of log files, file checksums, file and directory permissions, local network port activity, and other basic host security items.  HIDS are generally required for compliance with HIPPA, PCI-DSS, and various “best practice” standards such as the ISO 27000 series.

HIDS offer the benefit of being able to detect attacks local to the machine or on an encrypted or switched network where a NIDS might have issues.

HIDS provide a wealth of forensic data and can often determine whether or not an attack, originating from the local host or the network, succeeded or failed.

Some disadvantages to HIDS include the administrative and computing overhead of running them on each machine, and the potential compromise of the HIDS in the event of a successful attack.  HIDS can generate a surprisingly large number of alerts, and if you have dozens/hundreds/thousands of hosts to monitor, you will need to have a centralized alert DB and analysis tool.  Finally, some HIDS are licensed per host, making the costs of HIDS infeasible for every host when your organization has hundreds.  Still, there’s not law that you must monitor every host; you can deploy HIDS on just the most important and/or vulnerable hosts.

In contrast, network-based IDS (NIDS) monitor what traffic they can see.  This allows detection of attacks that target more than a single host, such as nmap scanning of a network, looking for hosts.  NIDS usually have a number of well-placed sensors plus a central management console/server.  Getting the traffic to the sensor can be difficult.  Usually, a tap is used in high-volume locations.  In other cases, you attach the sensor to a span (or mirror) port of a switch.  Such sensors can be simple hosts (stripped down VMs), which passively listen for packets using an unnumbered NIC in promiscuous mode.

While NIDS can scale up economically, they lack the ability to examine host changes.  Often, NIDS can’t tell if an attack was successful or not.  If data is encrypted, only a HIDS can examine the decrypted network traffic.

Sensor location critically affects NIDS abilities and costs.  At one end, you could deploy a sensor in front of each and every host.  This would give you all of the problems of HIDS, with none of the benefits.  At the other end, you could have a single sensor between your ISP and organization.  Such a sensor could be easily overwhelmed with too much traffic to analyze in real-time.  Additionally, some traffic can bypass such sensors in some cases.  Whether you deploy one sensor per subnet, or have fewer sensors, is up to you.

An intrusion prevention system (IPS) is the same as an IDS, except that instead of merely alerting, they actively block perceived attacks, usually by sending updating blocking rules to one or more firewalls.  A danger of IPS is blocking legitimate traffic, either by false positives, or by an attacker triggering the blocking as part of a DoS attack.

The sensor of a NIDS is configured in an unusual way.  If you enable a firewall on that host, most of the traffic the NIDS must see gets filtered.  So you need to turn off the firewall on that host.  But that is dangerous, so always strip such a system to the bare-minimum and harden that as much as possible.

Because of the large number of alerts that may be generated by an IDS, some sort of central DB and analyzer is used.  One popular commercial tool is splunk.

HIDS Types and Operation

HIDS (Host IDS) are generally File Alteration/Integrity Monitors (FAM/FIM).  Such tools run periodically from cron and look for altered files, such as hackers modifying important files and infecting programs with viruses.  This is done by computing checksums/CRCs for a set of files (generally you don’t monitor /home or /var, as you expect files there to change all the time).  When run, the system re-calculates the checksums and compares them with the previously saved checksums.  The saved checksums are digitally signed and ideally stored on a read-only media such as a CD-ROM (or kept on another host).  Tripwire and Samhain are popular HIDS that work this way (FAM/FIMs).

Other types of HIDSs are possible, that examine log files and audit records, looking for suspicious activity, or more traditional virus scanners.  Such HIDS are signature based.  Samhain for instance checks for file checksums and detects modifications, searches for rogue SUID executables, detects rootkits (under Linux and FreeBSD), and is able to log to an SQL database.

Finally, there are statistical anomaly based (SAB) detectors.  This type of HIDS monitors system (and/or library) calls, looking for unusual actions (e.g., httpd trying to set the date or open a listening socket for port 22).  This is known on *nix as syscall monitoring.  No popular open source HIDS is statistical anomaly based as far as I know; Microsoft has made available in 2015 its Advanced Threat Analytics (ATA) tool for windows, which is SAB.

OSSEC HIDS

The most popular and powerful HIDS by far is OSSEC.  [From OSSEC in a Nutshell] OSSEC is a popular host-based intrusion detection system (HIDS) that is an open source project owned and sponsored by Trend Micro.  The OSSEC users include Netflix, Samsung, Apple, Barnes & Noble, NASA, and others.  Use OSSEC to monitor system logs, do file integrity checks, look for rootkits, check for registry changes (on Windows systems) and take actions based on security events that are detected.

OSSEC operates as an agent-server system.  Agents handle monitoring logs, files and (Windows) registries then sending back relevant logs in encrypted form to the OSSEC server over UDP port 1514 (default port).  The agents are small and run in chroot jails (on *nix systems).

On the manager host, the logs are parsed with decoders and interpreted with rules that generate security alerts found in the log stream.  OSSEC comes with a rich set of decoders and rules to track important system events, such as file changes, root logins, and much more.  Users can add custom decoders and rules to monitor any files and generate alerts specific to their needs.

[From alienvault.com]  OSSEC is very good at what it does and it is extensible.  OSSEC will run on all major operating systems.  It uses a client/server based architecture which is very important in a HIDS system.  Since a HIDS could be potentially compromised at the same time the OS is, it’s very important that security and forensic information leave the host and be stored elsewhere as soon as possible to avoid any kind of tampering or obfuscation that would prevent detection.

OSSEC’s architecture design incorporates this strategy by delivering alerts and logs to a centralized server where analysis and notification can occur even if the host system is taken offline or compromised.  Another advantage of this architecture is the ability to centrally manage many agents from a single server.

Most other open source HIDS are no longer popular as a result, except for samhain which is still actively maintained (2015).

There are several simple FAM/FIMs as well.  Most these are better than Tripwire in configurability, documentation, and the ability to reduce the logging output to a manageable level.  However, most alternatives to Tripwire lack strong cryptographic protection.  AIDE (Advanced Intrusion Detection Environment) is free FAM/FIM software that in theory does everything Tripwire can and more.  The integrity database is defined by writing regular expressions in the configuration file.  Besides MD5, several other algorithms are supported such as SHA1.  More recently maintained than AIDE is afick, which is reported to be good as well.

An unusual HIDS is PortSentry.  It monitors network traffic at a host, listening for signs of port scanning. It can modify a firewall automatically to block the source IP of the scan.  PortSentry can be used in conjunction with other HIDS.  PortSentry is part of the open source sentry security tools, along with logcheck and logsentry, which monitor log files and provide alerts.  PortSentry remains popular (2014) even though there’s been no active development for many years (since 2006-ish).

Under the Solaris system, administrators can use BART, the Basic Audit Reporting Tool, to track files at a system level (it’s a FAM/FIM).  On earlier versions of Solaris, checksums for system files can be matched against the Solaris Fingerprint Database (needed a SunSolve account, which I had).  While neither of these is a full-blown HIDS, they do provide basic functionality for detecting modified files.

rpm -V packagename is a pseudo-HIDS on RPM based systems.  (Try: vi /etc/DIR_COLORS; rpm -V coreutils)  This shows what has changed for each changed file in the package: 5=MD5, s=size,T=mtime, ... (See the rpm man page for “VERIFY OPTIONS”.)  For Debian Linux, use the debsums utility.  For *BSD systems, see mtree(8).

As of Linux 3.7, some HIDS functionality has been included in the kernel itself.  The Linux Integrity Measurement Architecture (IMA) was initially added to 2.6.30.  The full functionality depends on the presence of the Trusted Computing Base modules, based on the standards from the Trusted Computing Group.  Basically, the kernel will digitally sign a file’s hash code and store it in the file’s extended attributes.  The file’s integrity can then be checked on every access.  Note the kernel can also sign and verify LKMs (supports Secure Boot).

Using Tripwire HIDS

Download Tripwire from sourceforge.net/projects/tripwire.  Unpack in $HOME.  Read the INSTALL file and the ./configure --help output to choose your options.  I used:
   ./configure --sysconfdir=/etc/tripwire --enable-static

This didn’t work!  A visit to the website “help” forum, there was a thread for Fedora (FC5).  Turned out Tripwire had a known bug and a patch was available.  Download the patch to $HOME and run it with patch -p0 <patch-file.

Next cd into the tripwire directory and run configure again, then make clean and then make.  Even with the patch an “su -c 'make install'” didn’t work.  Apparently the contrib directory should have been named install (?).  After creating a symlink with the correct name, it finally finished the make successfully.

There are a few post-install tasks needed:  cd into the contrib directory and edit the cron script tripwire-check to change the “/usr/local/etc” path(s) to “/etc”.  Finally, as root, you must copy this script to /etc/cron.daily (or put it elsewhere and leave a symlink to it, if your cron allows symlinks to crontab files.)

After installing, you must configure it.  Besides setting various options the way you like you need to define your system policy (what will be monitored by tripwire).  In /etc/tripwire you will find twcfg.txt, the configuration file, and twpol.txt, the policy file.  The default configuration is probably okay but you may wish to turn on syslog reporting instead of email, or play with the log level.  Setting LOOSEDIRECTORINGCHECKING to TRUE will not report changes to a watched directory itself, only the contents, which is usually good enough (else you get two reports when a file changes!)  See man twintro(8).

Tripwire Policy File   (See man twpolicy(4) )

Creating the policy is the hardest part.  It is also the part that needs updating when your system configuration is updated (once a year or so on a typical production server).  The policy file has two parts.  The first sets file locations and values of directives.  The second part is a (huge) list of rules: files (filesystem objects, which include directories, named pipes, sockets, and devices) to watch and what attributes to watch (and ignore).  A rule for a directory means that directory and everything in it (recursively).

object_name  white-space  ->  white-space  property_mask [(attr=val[,...])] ;

The property­_mask is a list of the form: +xxx-yyy to check xxx but not yyy.  If some property is not mentioned, it means to not check it, so you can often omit the “-yyy” part.

Because some combinations are frequently used in your policy for different objects tripwire allows you to define a variable for that can be used for the property_mask (or even an object name):

          name white-space = white-space value ;

You use these similar to shell variables:  $(name).  A number of variables are predefined by Tripwire: ReadOnly, Growing (for some log files), Device, IgnoreAll (just make sure the object is there), IgnoreNone (useful with: $(IgnoreNone)-yyy), and some others.

Here are some examples:

mask1 = ...;
mask2 = ...;

# Scan /etc directory tree using mask1, except the
# file /etc/passwd, which is scanned using mask2:
/etc        ->  $(mask1);
/etc/passwd ->  $(mask2);
/lib -> $(ReadOnly) (emailto = bob@foo.com, severity = 80);

Only one rule may be associated with any given object.  If any object has more than one rule in a policy file, Tripwire will print an error message and exit without scanning any files.  For example:

# This is an example of an illegal construct:
/usr/bin   ->   $(mask1);
/usr/bin   ->   $(mask2);

Other things can go into a policy file, including directives (e.g., “email this violation to wpollock”, severity numbers (these can be used in the report, with a Perl script to handle severe violations differently), section definitions, and other stuff.

If a bunch of attributes apply to many rules, rather than repeat them per rule you can use this syntax:

( attribute-list ) [\n] { \n rule; \n rule; \n ... }

You need to decide which parts of your system to monitor, and to what degree.  A simple basic policy can be used to protect only your critical files.  For example binary directories and their contents should never be modified without detection: /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/[s]bin, /opt/[s]bin, and so on.  (Basically the directories found by running:
    find / -type d -name \*bin

You must also protect any lib directories, as these contain kernel modules, PAM modules, and DLL files.

Next you should protect configuration files.  On a learning host like ours you should expect most of these files to change regularly, but rarely in a production setting.  These directories include /etc and other directories in /usr, /opt, etc.  Additionally some programs put config files in strange locations, such as under /var/lib/someplace.

In /etc are a few special files that are expected to be present, but whose contents changes all the time.  Examples include mtab and ld.so.cache.  If you use a general policy rule for all of /etc and its contents (a good idea), you must make exceptions of such files.  (mtab can be a symlink to /proc/mounts on many systems, and ld.so.cache could be put under /var, with a symlink left in /etc.)

Log files should exist and not change, except to grow or be rotated (which usually changes the inode number then shrinks the file).

Finally you need to protect other important but not critical files, such as timezone files or printer configuration files, often found under /usr someplace.  Also, on a production machine you are rarely adding users, so the directories under /home may be usefully monitored, although a looser policy to allow legitimate users to add or change files should be allowed (perhaps it is enough to monitor system account home directories).  Also it may pay to monitor /root, but to allow some files to change without reporting them (e.g., .bash_history, .viminfo).

Tripwire Administration

Once you set the configuration and policy files the way you like, you then generate some encryption keys, one (the site key) used to encrypt the configuration and policy files and the other (local) to both encrypt and sign the tripwire database.  If you plan to run tripwire unattended via cron you might consider using no password on the local key.

To create the passwords run (as root):

tripwire-setup-keyfiles

(or you can use “twadmin --generate-keys”.) 

When run for the first time, a database of all the digests/hashes is built from all the specified files.  This is then digitally signed and encrypted, to make sure it isn’t changed by some attacker.  Ideally you would burn this DB on a CD-ROM.  The command for this is:

tripwire --init

To run a check on your system and have tripwire generate a report, run:

tripwire --check

At this point you may discover some flaws with your policy file.  Now is a good time to fix those.  Edit the policy file and then run

tripwire --update-policy

to update the policy (the encrypted version) and to update the database to match the new policy.  Run a check again and repeat until you have a good policy.

Now is a good time to back up the encrypted policy and configuration files.  Then delete the plain text versions.  You can get these back using:

twadmin --print-cfgfile
twadmin --print-polfile

You may also want to back up the database; burn all three files on a CD-ROM.

Then (via cron) tripwire runs and checks all the watched files, recomputing their hashes and comparing them to the ones saved in the DB.  This may require you to enter a password to read the DB.  If that is annoying you can remove the encryption from some or all of these files using the command:

twadmin --remove-encryption ...

Eventually you will see a report showing something has changed.  If you (or another administrator) didn’t update the software and it is several important files (e.g., /sbin/login, passwd, ...), now would be the time to suspect a rootkit and/or virus has replaced those binaries.  You should take appropriate action (including restoring the bad files from a backup).  If the change is harmless (a result of running yum for instance) you need to update the database with the new information or you will see that error in all future reports.  You can do this in one of two ways:

tripwire --update [-a]

With the “-a” option the DB will automatically update the database with the current system data.  Without it, tripwire will launch the editor on a text file.  In that file there will be an “x” next to each reported violation.  Leaving that “x” causes that item to be updated, removing it causes no update for that item.  When you save and quit the editor, the DB will be updated.

Use the twprint command to produce human-readable copies of saved tripwire reports.  Note if these are saved (e.g., in /var/log/tripwire) you may need to rotate them over time.

NIDS Types and Operation

NIDS (Network-based intrusion detection systems) run on one or several carefully placed hosts (called network monitoring stations or NMS) and view the network as a whole.  NIDS use NICs running in promiscuous mode to capture and analyze raw packet data in real time.  They need to be configured for your organization’s normal traffic to reduce false-positives.  (Many SAs omit this vital step!)

Most NIDS (and HIDS too) use one of two methods to identify an attack: statistical anomaly detection or pattern-matching (or signature-based) detection.

A signature based NIDS such as snort examines visible network activity, looking for suspicious activity such as port scans, packets with fake source addresses, etc.  Not all malicious activity is visible on a given host’s NIC, so the location and number of such scanners must be carefully thought out.

An anomaly based IDS looks for activity that is different from a user’s or system’s normal behavior.  (Credit companies use the same technique, to spot unusual credit card purchases.)  This approach self-trains to create a baseline.

Anomaly based IDSs causes many more false-positives than with a signature based IDS, thus requiring more effort over time.  But they can detect zero-day attacks that signature based IDSs can’t.

Snort is the most popular NIDs (2015).  It is signature based. Another very good one is Suricata.  While Suricata is designed differently than Snort, it is also signature based and can in fact use Snort rules.  Suricata is multi-threaded and can use hardware acceleration (i.e., a GPU), can capture malware or other designated traffic in files, and can have rules that use more than packet data.  Suricata also provides nice dashboards and is easy to use.

Bro (Bro-IDS) is the only popular open source NIDS that is anomaly based.  While probably the most powerful tool, it is considered difficult to learn and use.

Anti-Virus Scanners, malware scanners

In addition to HIDS that look for modified files, scanners check downloaded data such as downloaded packages and most commonly, email messages.  These help prevent the system from getting infected (modified) in the first place.  Note web and other servers can spread various types of malware, including viruses, trojans, worms, cookies (a type of spyware), etc., even if your system itself remains un-infected.  So-called adware is another form of malware that can be blocked.

There are a number of ways these scanners can work, but most commonly, they are signature-based.  Most only work for Windows (or scan for Windows malware, even when run on a *nix host).  Examples of common anti-virus scanners include Clam AV (clamav.net), Panda (pandasecurity.com), ESET NOD32 Antivirus 4 (eset.com), AVG Anti-Virus (free.avg.com), and Norton AntiVirus us.norton.com/antivirus).  There are scanners for Android and other mobile devices too.  These generally also offer GPS location, backup, remote-wipe (in case of theft).  (I like Lookout, but most are about the same in cost, features, and reliability, so it probably doesn’t matter much which one you use.)

Signature-based scanners are about 93% accurate, but that number is misleading.  They detect most of the more common malware (closer to 100%), but do less well against new or rarely seen malware.

Vulnerability Scanners

There are a class of related tools known as vulnerability scanners or security audit tools that look for potential problems in your host (and in some case, network) configuration.  Examples include bastille, nessus, saint, and others, listed previously.  Some scanners such as Nagios will also work as an IDS.

The Center for Internet Security (cisecurity.org) provides excellent vulnerability scanners which they call benchmarks, some for free trial use (CIS is non-profit but they do charge for some tools and support).  The platform specific scanners provide a report and include generally excellent security advice in the accompanying documentation.  (CIS_Solaris_Benchmark_v4.0 includes an 89 page security document that is one of the best security resources available.)  CIS benchmark scanners are available for all Unix, Linux, and Windows systems, plus a few network devices (e.g., Cisco routers) and major applications (e.g., Major DBMSs, Exchange Server, BIND, Apache, etc.).

Sun also makes a security vulnerability scanner tool for Solaris called “JASS” at www.sun.com/software/security/jass.  (The name came from JumpStart Architecture and Security Scripts, but is now called Solaris Security Toolkit.  The acronym is still JASS for some reason.)

Lecture 11 — Security Assessments, Evaluations, and Audits, and ROI

Qu: How do you know you are secure?  The answer is you must conduct assessments and evaluations to know.  Being secure doesn’t mean your system can’t be hacked or compromised.  There is always a risk!  (“Prevention eventually fails.”)  To be secure means you have an acceptable level of risk.

Assessment

A Security assessment is the first step.  The assessment involves the whole organization and assesses the non-technical aspects of security.  It is useless (or worse, leading to a false sense of security) to evaluate the technical aspects before you know what must be protected.  For example, evaluating the network encryption of data is pointless if the data is easily obtained from a non-protected database.

The assessment should examine the security policies (and related policies, such as disaster recovery policy or DRP), procedures, organizational structure and infrastructure architecture.

However a preliminary step is to determine exactly what resources are considered vital to protect.  You must know what the organization considers valuable and/or private.  If possible dollar amounts should be assigned.  (This step is related to the risk analysis done for a DRP.  Note you can often get help assigning dollar amounts from organization’s finance/insurance dept.)

Next you need to make sure the policies, procedures, organization, and infrastructure, if all implemented correctly, would protect that data to the levels required.  Often, an organization will adopt inappropriate policies and procedures (sometimes self-contradictory ones).  It doesn’t matter how well you have implemented security measures if they wouldn’t protect the data that needs protecting!

You must also make sure the policies and procedures are in compliance with applicable laws, regulations, and best practices.  Examples include HIPAA (Health Insurance Portability and Account Act), SOX (Sarbanes-Oxley act of 2002), GLB (Gramm-Leach-Bliley act), DMCA, FMA (Financial Management and Accountability act), FERPA (Family Education Rights and Privacy Act), CFAA (Computer Fraud and Abuse Act), and many other federal and state regulations and laws.  International regulations may apply as well, such as the EU’s General Data Protection Regulation (new in 2018).

Many of these laws and regulations require the data they use to be available, thus essentially mandating a good disaster recovery policy.  (SOX 2002 is often cited for this.)

Return on Investment (ROI) [Adopted from Network Security Evaluation
                                                   by Cunningham et. al, (C)2005 Syngress, p. 57]

A quality assessment of a medium to large sized organization could take months for a (small) team.  For this reason such assessments (like risk analysis) are usually left to consultants who will receive tens of thousands of dollars for the work.  To justify this expense the system admin should be able to explain the current and expected return on investment (ROI) or ROSI (return on security investment) for the assessment and/or evaluation.  (And indeed, for any security related budget.)

Unfortunately there isn’t a single widely agreed upon method for this calculation.  (Do a Google search for “security ROI”.)  Traditional ROI calculates an actual payback and/or reduction in expenses.  But for security money spent you will rarely if ever see such a return.  Instead you must focus on security expenses as a kind of necessary insurance, and the ROI is the lowered risk exposure.

Understanding Risk [from:  www.csoonline.com/read/120902/calculate.html]

According to one study the American Society of Safety Engineers (ASSE) cites, the ROI of fire extinguishers is in fact about a $3 return for every $1 invested if you take fire extinguishers as part of a larger corporate health and safety initiative—which you should, since fire extinguishers (like IT security) rarely show up as a discrete security purchase.

... A binary thinker might suggest that, since there was no fire last year, there was no ROI.  If that is the attitude at your company, it’s time to initiate some awareness and education because that’s not how risk mitigation works.  Think of it this way: If you wear your seat belt but don’t get in a car accident, does that mean you ought not to invest in a seat belt because there was no return?

No.  You did get a return, because return is not measured in a dogmatic world of what did or did not occur, but in the stochastic world of what might occur and how likely it is to occur.  That is the game of risk; prepare for something to happen by investing in ways to stop it from happening.

Calculating ROI

One way the security ROI can be estimated is by computing the annual estimated loss expectancy of a security incident, which can be calculated as:

          Incident-cost x probability x mitigation + cost-of-mitigation

Note a mitigation is some measure taken that either reduces the probability of an incident, or reduces the cost of recovery should some incident occur.  The annual estimated loss expectancy is the sum of these calculations for each incident, for some combination of mitigations.

Suppose the cost of a security break-in $1,000,000 and the probability of this is 35% if you do nothing.  Then the annual loss expectancy if you take no action is $1,000,000 x 0.35 = $350,000.

Next consider what mitigations you can take.  Each has a cost so you need to determine which mitigations are the most cost-effective, given the limitations of your overall budget.

Suppose a $20,000 anti-virus program might reduce the chance of occurrence by 50%, a $10,000 user security awareness training program might reduce the expected loss by (say) 33%, and a $15,000 security assessment and evaluation by 35%.  The choices are:

Of course you need some reliable industry data to back up these claims for improvement.  (The numbers used here are unrealistic.)  Also, you can combine various mitigations to reduce the expected loss (and hence increase the ROI).  Also in a real calculation, you would need to consider initial or one-time (capital) costs (and depreciation) and recurring (operational) expenses.  Qu: if your annual budget was $25,000, what mitigations would you recommend?

“Building security into software engineering at the design stage nets a 21% ROSI.  Waiting until the implementation stage reduces that to 15%.  At the testing stage, the ROSI falls to 12 percent.” – From: www.cio.com/archive/021502/security.html

For the loss that is left after all mitigations, you can get cyber-insurance.  Some sources include: Lloyd’s e-comprehensive, Chubb’s Cybersecurity, and AIG’s Net Advantage Security.

The NSA (National Security Agency) has published the Information Assurance Methodology (IAM), with practical and very valuable advice on how to conduct an assessment, including for example how to negotiate the contract for the consultants.  (Show book.)

Evaluation

Once you have determined the correct procedures, policies, and infrastructure required to provide adequate protection, it is time to conduct a Security Evaluation.  In this phase you examine the equipment and configurations to make sure your routers, switches, firewalls, servers, and hosts will implements the policies determined by the assessment.

The NSA has also published the Information Evaluation Methodology (IEM) as a guide to conducting the evaluation.  (Show book.)

The definitions of assessments and evaluations as explained here are not universally agreed upon terms.  Many would consider these terms interchangeable with each other, and also with an audit. 

Audits

The main goal of IT audits, assessments, and evaluations is to reduce business risk and liability exposure.  IT has become a critical part of an overall business strategy.  The audit does this by finding abuses, security problems, compliance violations, and other configuration errors that can lead to poor performance.  Today most people speak of a security audit, which is only concerned with locating (and later fixing) security related issues.

An Audit may be defined as a comprehensive examination and evaluation of a system (possibly a network).  An auditor examines the configuration of the system and past and current activity as recorded in various log files.  Some extremely detailed log files are called audit logs or audit trails, and/or system activity logs.  Usually such logs are not enabled except when an audit is in progress.  These logs can report every action of every process (most audit logs only report the OS system calls made by each process, which is usually good enough).

There are many types of audits, even just considering IT audits.  Once type of audit is a routine part of on-going security measures (this can be called an assessment/evaluation).  Another type is performed after a problem has been discovered (or suspected), to assess what is wrong with the policies and procedures and to assign blame.  A security audit is conducted to trace a series of related events, to determine the trail of an attack.  This often involves examining multiple logs, sometimes from multiple systems, and sometimes working backwards from the final event toward the initial event of the attack.

An organization may conduct internal assessments and evaluations to ensure to themselves that the system works correctly and efficiently.  Audits are conducted to provide the same assurances to shareholders, business partners, customers, and government regulators (external audit).

Auditors ensure:

·       An adequate audit trail exists for all activities, especially financial transactions.

·       Controls to ensure data integrity.

·       Both unit (subsystem) and integration (whole system) testing procedures, to make sure the system performs as intended.

·       Exceptional conditions are anticipated as much as possible, and general exception handling procedures are in place.  This includes authorization procedures for system overrides when needed.

·       Controls exist to prevent unauthorized and/or undocumented changes to the infrastructure (and indeed, to any procedures).

·       Required government, industry, and internal policies and procedures are followed.  (Sometimes referred to as compliance audits.)

·       Adequate training is provided for personnel.

·       Additional security, compatibility (different vendor equipment), ..., and controls.

Audits may be conducted to:

·       Ensure confidentiality, integrity, and availability of info and resources.

·       Ensure compliance with various regulations and policies (e.g., SOX, HIPAA, HITECH).

·       Ensure conformance to the local security policies.

·       Investigate possible security incidents.

·       Monitor user or system activity where appropriate.

Auditing is sufficiently hard that most organizations will hire out the audit.  Auditing doesn’t need to be done continuously but should be done periodically, especially after a system upgrade.  (See The Unix Auditor’s Practical Handbook.)  Compliance and legal issues may force an audit (of some sort) every year.

A basic audit log can be made by running an IDS frequently (continuously, but may be hourly, daily or weekly runs are enough), and regularly running a vulnerability scanner.  However a complete audit could take days to weeks (or even months!) to complete and is expensive.  These may only be run annually, or only after a major change to the network or some server.

Audits have a negative connotation for many organizations.  They are viewed as a “fault finding” expedition when something has gone wrong.  For this reason proactive audits are often called “assessments” and/or “evaluations”.  From a security standpoint, an assessment is used to determine what policies are required to protect valuable information assets.  An evaluation is used to determine if the procedures and configurations (of firewalls, routers, switches, and servers) are correctly implementing the policies.  (As per NSA.)  As a result, these terms are often used interchangeably.

Solaris uses BSM for auditing, enabled by starting the audit daemon (auditd).  The /etc/security directory contains subdirectories with audit files related to audit control.  auditd collects data and writes it to records, forming audit files.  To check if BSM is running, type “ps -ef |grep auditd”.  If not, research bsmconv on Solaris and ausearch on Linux (“man -k audit” works too.)

Audit files that are complete have names like: start-time.finish-time.hostname.  The exact format is: YYYYMMDDHHMMSS.YYYYMMDDHHMMSS.hostname

For example: “20020610020543.20020617225351.SERVER01” indicates the log file on Server01 was started June 10th, 2002 at 5 minutes after 2 am, and closed itself successfully on June 17th, 2002 at 10:53 pm.

If the audit log file is still active or was interrupted, it uses the form: start-time.not_terminated.hostname.

Enabling auditing adds a 1% to 3 % performance penalty, not very much!  However the audit logs can grow by many gigabytes a day; you must allow for that.  OS auditing tools collect data that makes it possible to:

·       Monitor security-relevant events that take place on a host

·       Record the events in a network-wide audit trail

·       Detect misuse or unauthorized activity

·       Review patterns of access and the access histories of individuals and objects

·       Discover attempts to bypass the protection mechanisms

·       Discover extended use of privilege that occurs when a user changes identity

Some important IT related audits include:

C2 Audit:  Many Unix systems allow the admin to enable auditing called “C2 audit” which is the form specified by Department of Defense regulations to meet C2 level of trust.  (DoD 5200.28-STD, “Department of Defense Trusted Computer System Evaluation Criteria” (TCSEC), commonly known as the orange book.  This was part of the rainbow series.)  C2 auditing generally means assigning an audit ID to each group of related processes at login, and thereafter related system calls performed by every process are logged with that audit ID.  There is no accepted standard for the logging format; C2-style logging can have different formats, controls, and locations.  C2 audit is useless without interpretation and reduction tools, such as those provided in the Solaris Basic Security Module (BSM).  (Linux doesn’t call it that.)  It doesn’t work well for networked systems.

Common Criteria (CC):  This is an international standard (ISO/IEC 15408) for computer security (and is available for free download, e.g. from niap-ccevs.org).  This has replaced the TCSEC.  The Common Criteria does not provide a list of product security requirements or features that products must contain.  Instead, it describes a framework in which computer system users can specify their security requirements (called protection profiles), from a common list of criteria.  Vendors can then implement and/or make claims about the security attributes of their products (that is, which protection profiles they meet), testing laboratories can evaluate the products to determine if they actually meet the claims.  The labs are audited by governments (in the U.S by NIAP CCEVS, part of the NSA) to ensure they use the testing procedures listed in the CC.

SAS70:  Statement on Auditing Standards (SAS) No. 70, Service Organizations, is a widely recognized auditing standard developed by the American Institute of Certified Public Accountants (AICPA).  Released in 1992, SAS70 is authoritative guidance that allows service organizations to disclose their control activities and processes to their customers and their customers’ auditors in a uniform reporting format.  This is especially useful for on-line companies, to provide customer confidence in their data handling procedures.  In 2010, SAS70 was updated and renamed to SSAE 16, but most still refer to SAS70.  (The new standard is available since 6/2011.)  SAS70 is being revised to reflect the same changes made in SSAE 16, so both are essentially the same standard.  A company such as Box.net that is SAS-70 certified, allows admins for enterprise accounts to run full audit trails of all users.

Safe Harbor and Privacy Shield:  While all governments want to protect their citizens’ privacy (or so they claim), the European Union (EU) was the first to pass comprehensive laws, in 1998 (“Directive on Data Protection”).  This causes a problem for multi-national companies, since non-EU members may have different laws in place.  In order to bridge these different privacy approaches, the U.S. Department of Commerce, in consultation with the European Commission, developed a “Safe Harbor” framework.  Organizations can evaluate and possibly join the U.S.-EU Safe Harbor program.  (There is a similar agreement for the U.S. and Switzerland.)  Members are claiming to comply with both sets of laws.  Note, these programs are self-certified.

In 2015, after complaints and evidence of PII being used inappropriately, the EU decided to end their safe harbor agreement with the U.S.

In 2016, the EU approved a sucessor to Safe Harbor known as Privacy Shield.  This is similar to Safe Harbor but with more transparancy, oversite, and a redress procedure for EU citizens (missing from Safe Harbor).

PCI (Payment Card Industry) Compliance and Security, Consumer Privacy

The Payment Card Industry (PCI) has created requirements known as the PCI data security standard (or DSS) for protecting payment card information, including information in computers which process, store, or transmit (across the Internet) credit card and other payment card information.  Some non-obvious issues include data retention issues and data disposal issues.  Many commercial organizations need to handle such electronic payment information.  Merchants and payment processors need to regularly file notices of compliance, usually created by an internal audit.

In the past each type of card (Visa, Amex, ...) had security regulations for handling customer names, addresses, and other collected payment information.  (E.g. usa.visa.com/merchants/risk_management/cisp_merchants.html.) Recently all the major banks have agreed to a single set of standards, the PCI DSS.  For example, under the new PCI standard an un-encrypted log file is not allowed to store a complete credit card number; but you can store the first 6 and last 4 digits.

Credit Card and Electronic Payment Information: Fraud Protection

There are two mechanisms currently in place in credit card technology that assist a merchant in minimizing credit card fraud.  Both of these are of particular benefit when processing credit card transactions “anonymously” as they are done online.  These two mechanisms are nicknamed “AVS” and “CVV”.  For your own protection as a merchant, you are strongly encouraged to require both (or at least CVV) from your customers.

AVS stands for Address Verification Service.  This is a means by which you can determine if the address the customer provides agrees with the address associated with the credit card account.  This provides you with some measure of confidence that the person using the credit card is the person who owns it.  AVS information is not required for credit card transactions.  However, if it is provided, the transaction processor issues back a response indicating how much of the address matches.  There are levels of matching: it may be the case that the street address is correct, but the postal code is wrong, etc.  It is important to note that a credit card transaction will not usually be denied if the address is wrong.  It is up to you to decide what to do if the address does not match or matches only partially.  One course of action would be to void the transaction and deny the sale, but this is entirely at your discretion.

CVV stands for Card Verification Value (sometimes called CVV-2).  This is a 3 or 4 digit code that is found on Visa, MasterCard, Discover, and American Express cards.  It appears on the card but nowhere else, not even on credit card statements.  Its primary purpose is to validate that the user of the credit card actually has the card in hand.  On Visa, MasterCard, and Discover CVV data consists of the 3-4 digits found after the card number on the signature strip on the back of the card.  On American Express CVV data consists of 3-4 digits on the front of the card, to the upper left of the card number.  CVV data provides the merchant a high degree of confidence that the customer actually possesses the credit card that is being used.

Consumer Privacy Protection

Besides protecting your enterprise from credit card fraud, it is important to guarantee customers that their credit card information is protected.  Of course you should use encryption for you web site (HTTPS rather than HTTP for all web pages that require security).  But there are additional protections needed too.  There are at least four different times when customer credit card information is being transferred and must be protected (there may be more depending upon how your business operates):

1.    When the customer sends his/her credit card information to you.  When processing credit card transactions online, this step takes place over the Internet, and guaranteeing security at this time is entirely your responsibility as an Internet merchant.  You must have a secure server, with a valid security certificate, and you must be sure to use the “HTTPS” protocol whenever your customers are sending private information.

2.    When the credit card information is sent to the transaction processor.  You do not have direct control over security at this step, but you will need to be sure that the credit card transaction processing software you use is secure.  All credit card information should be securely encrypted as it is passed back and forth from your server to the credit card transaction processor.

3.    When credit card information is being moved into and out of a database.  Again, the security measures in place at this step will depend upon the software you are using to handle your transactions and maintain the database.  Whatever credit card transaction processing software (CCTPS) you decide to use, never store unencrypted credit card information in a database.

4.    When credit card information is being viewed and handled by your staff.  Ensuring security at this step is your responsibility. Some CCTPS includes a reporting capability that will be of great use to your accounting personnel.  Since these reports include credit card information, you will need to be sure to restrict access to the site that hosts the CCTPS and the DB, and to run it on your secure server.

For more information see pcianswers.com and www.pcisecuritystandards.org.

Privacy Assurances

Two further measures should be taken by merchants that offer on-line sales.  The merchant’s site must have a privacy policy that should meet the P3P Privacy Policy standard.  (See www.w3.org/P3P/).  While mainstream browsers don’t currently include support for this, there is no competing standard to use.

Secondly your site should include some assurances to the user about your on-line business practices (especially data collection, retention, and handling).  This is commonly done by having a well-known third party certify your site as compliant with their policies and practices. (This is in addition to the PCI DSS.)  You include a special icon and notice of compliance on your site.

Remember that few companies are so well-known that assurances are not needed.  Some common assurance providers are BizRate.com, ResellerRatings.com,  TRUSTe.org, BBBOnLine.org, TrustWave.com, and VeriSign (verisign.com/trust-seal/).  (Show the bottom of the welcome page at NewEgg.com.)  Many of these are based on consumer reviews.

Two tips for storing audit trails: either compress it using gzip or save them on a file system with compression enabled.  If you are required to audit all file access it will generate a gargantuan amount of data, probably several gigabytes per day per system.  You need to minimize this, without post-processing the audit trail if possible.

There are two sets of files for which we must audit access to:  cardholder data and audit files.  If you make the cardholder data owned by a role (e.g. “pci”), and set the file mode so that only the role may access the file (chown pci and chmod 0600), you don’t have to audit fr for everyone.  It will be enough to audit fr for the pci role.  When the users who are authorized to access the data assume the pci role, they get audited for fr even though their normal account aren’t.

However, since root can read all files that account also needs to be audited for fr.  This also takes care of the auditing of access to the audit files, which are only accessible by root.

To catch someone changing the mode of cardholder data (e.g. making it world readable) the pci role should be audited for +fm (successful file attribute modification).

Solaris Audit configuration files:

Below is the audit configuration for the PCI DSS version 1.1:

/etc/security/audit_control

flags:lo,-fr,+fc,+fd

/etc/security/audit_user

root:ex,fr,fw,+aa:no

pci:fr,fw,+fm:no

Lecture 12 — Process Privileges — Linux Capabilities and Solaris Privileges

Process Privileges — Linux Capabilities and Solaris Privileges

Traditional Unix implementations distinguish two categories of processes:  privileged processes (whose effective user ID is 0, referred to as super-user or root), and unprivileged processes (whose effective UID is non-zero).  Privileged processes bypass all kernel DAC permission checks, while unprivileged processes are subject to full permission checking based on the process’s credentials (usually: effective UID, effective GID, and supplementary group list).

This traditional Unix approach is flawed in that it violates the principle of least privilege: to be able to perform any privileged act (e.g., listen on a well-known port) a program must be enabled to perform all privileged acts (e.g., shut down the system or write to read-only files in /bin).

Originally the SUID feature of *nix was designed for this situation.  The idea is to make the data only accessible by app-owner (an otherwise unused user account with no login), make a SUID executable accessible (executable by all) owned by app-owner.  Now the only access to the data is indirectly via this “proxy” application, which in theory can implement an arbitrary security system.  The problem is this is extremely hard to do well and means every application uses a different authentication/authorization scheme.  Most implementations used SUID to root, to make configuration simpler (no extra permissions or groups to manage).  Today there are few SUID programs and most of these should not be SUID!

One commonly used approach is to use privilege bracketing.  With privilege bracketing, privileges are enabled only when they are actually required and turned off when they are no longer required.  (You see this used in daemons, which start as root, then “drop privileges” by changing the effective user ID (euid).  The idea of privilege bracketing is to narrow the window of a program’s vulnerability so that it is as small as possible, reducing the damage any exploit can do. (Ex: your FTP server runs as root; a remote attacker uses an FTP server security hole (vulnerability) to gain a root shell.)

A newer scheme that can be used in addition to privilege bracketing, is to provide a large set of individual privileges (Solaris), or capabilities (Linux, POSIX), where the god-like abilities available to processes whose effective UID is 0 are broken down into numerous discrete privileges. A process only is granted the rights it really needs.

A process can voluntarily remove some privileges but never add to them.  In this way, an error or security compromise is limited in the damage that can be caused.

Although using privilege bracketing and a minimal set of privileges/capabilities doesn’t reduce or remove software flaws that can be exploited, it does limit the damage that can be done.  However it does rely on the programmer to drop privileges; an SA can’t manage this security.  “sudo” helps with this.

Linux Capabilities

Linux (since v2.2) also has split rootly powers into discrete privileges, only in Linux they are called capabilities.  Each capability grants the process that has it the right to bypass some specific kernel permission check. (e.g., CAP_CHOWN, CAP_KILL).  There are close to 3 dozen of these.  (Show /usr/include/sys/capability.h)

A root process gets all permitted (/proc/sys/kernel/cap-bound) capabilities, which gives it traditional privileged status.

Any process can remove capabilities from its current (effective) set, but not add any that aren’t already in the permitted set.  This is one reason why daemons start as root, remove some capabilities (from effective and permitted), then switch to a regular user account.

Capabilities in a process’s current set can be marked to be inherited by child processes (similar to export).  The inherited capabilities of a parent process become the permitted capabilities of a child process (after a fork()).  Thus there are three sets of capabilities: permitted, inheritable, and effective (the ones currently set).  See man capabilities(7) for details.  Show cat /proc/$$/status, at the bottom is capability sets.  Show for init (PID of 1) too.  Then show getpcaps pid, part of lib[p]cap package; see cap_from_text(3) for an explanation of the output.  Also demo less /usr/include/linux/capability.h.)

Modern systems (such as FreeBSD and Linux) allow root to add some (permitted) capabilities to an executable program.  This can be considered the fine-grained version of SetUID to root.  See setcap(8) and getcap(8).  Demo both getcap /usr/bin/ping and getfattr -dm - /usr/bin/ping.

Solaris Process Rights Management

The Solaris 10 OS introduced process privileges, where the rootly powers available to processes whose effective UID is 0 are broken down into numerous discrete privileges.  This is very similar to Linux capabilities and is related to RBAC and Solaris containers (zones), discussed later.  Like Linux Solaris privileges are a superset of POSIX’s IEEE-1003.1e capabilities (Solaris 10 has about 60).

A privilege is a discrete right that a process requires in order to perform an operation.  The right is enforced in the kernel.  Programs can be run with the exact set of privileges that enable the program to succeed.  This is specified in a security policy.

For example a program that sets the date and writes the date to an administrative file might require the sys_time and file_dac_write privileges.  If that program is granted these privileges than it doesn’t need to be run as root.

To perform some privileged action the process must assert the relevant privilege first, using the setppriv system call.  A given process is allowed only a limited set of privileges it can assert.

For example, a process that needs to bind to a reserved port (i.e., the web server binding to port 80) used to require root privileges.  Now a process can do this by asserting the privilege PRIV_NET_PRIVADDR.  If a program has been written correctly other privileged operations such as the ability to read or write any file on the system will not be available to it.  The privileges man page describes each of the available privileges.  (There are also predefined privilege sets, e.g. All or Basic.)

A process owned by root has all available privileges, restricted by the privilege set assigned to the process’s zone, known as the “L” (limit) set of privileges.  The set of privileges a process is allow to assert is called its “P” (permitted) set, and the set of privileges a process has currently asserted successfully is called the “E” set.  Some of these can be marked for export, known as the “I” set (a subset of the P set, which in turn is a subset of the L set).  When a parent process creates a new child process (via fork), the child’s P set equals the parents I set.  With su to root, E = P = L.

Use the ppriv pid command to see these sets, and to start processes with reduced privilege sets:
ppriv -e -s L=basic ksh; ppriv $$; ppriv -lv

sudo and /etc/sudoers

Allows a sys admin to give rootly privileges to users without giving the root password out.  sudo provides a useful audit log; running stuff via su doesn’t.  Exactly what a given user is allowed to do (with rootly privilege) can be defined, although that can be tricky to get correct, and many systems (such as Ubuntu) don’t bother to try to restrict user actions.

Users run commands prefixed with sudo.  Sudo then checks the /etc/sudoers file to see if the current user is allowed to run the given command with the given args, from the (possibly remote) host they are logged in on.  The command then runs as root (sudo is suid).  Example of a user changing another’s password:

          /home/auser$ sudo passwd buser

sudo prompts users for their password and remembers it for about 5 minutes so you don’t have to re-authenticate for every command.  (It is easy to create a background job that runs some allowed sudo command every few minutes, to perpetually reset the timer.)

Every sudo action, whether sucessful or not, is logged (the file depends on your system; For RH, it is usually /var/log/secure).  (Demo.)

sudo is very popular on *nix systems.  On some Linux distros the root account is never used directly and all admin is done by prefixing commands with sudo (no password is asked).  On these systems root is not assigned a password, but I am not convinced this is a good idea!

The sudoers file must be setup carefully to allow authorized users permission to run their commands, but it is easy to accidentally allow too much access!  Edit /etc/sudoers only using visudo, which performs some sanity checks on the file before saving it to disk.  Man sudo, sudoers for details.  (Discuss /etc/sudoers example web resource.)

At the beginning of the sudoers file are a lot of definitions for aliases.  There are four kinds of aliases: User_Alias (list of users), Runas_Alias (list of usernames), Host_Alias (list of locations) and Cmnd_Alias (list of commands).  Such aliases can be used to define a list of users for example, so you don’t have to have many identical rules for each user.  A typical alias looks like this (note alias names must begin with an uppercase letter):

          User_Alias PWADMINS = user1, user2, user3

Aliases are very flexible.  A user alias can include users, groups, or even other User_Aliases.  Host_Aliases can contain IP addresses of hosts, network numbers (with netmask), or hostnames with wildcards (e.g., “*.hccfl.edu”).  A Cmnd_Alias can contain an absolute pathname to a command, a directory containing commands (e.g., “/sbin”), or the special word “sudoedit”.

The alias definitions are followed by a list of sudo parameter settings used to override the defaults.  For example, you can set certain users so they don’t have to authenticate to use sudo.  (That is the default for Ubuntu Linux, where any user can use sudo.)  You can define such parameters for any user of sudo, for all users logged in from a given location, or for a given user (no matter where they are logged in from).

At the bottom of /etc/sudoers is the heart of the file, the entries that define who can do what.  Entries look like this:

  user host = [(runas)] command [ argument(s) ] , ...

Or put another way:  who where = (as_whom) what

Each field can be a comma separated list or an alias (of the correct type).  Note a list can also contain aliases.  Using “!” means not allowed; this is useful in a list with wildcards to narrow the list.  For example the host list can be “*.example.com,!www.example.com”.

The first field is a username or a defined alias for a bunch of users.  You can also use numeric IDs.

The second field is the exact hostname as reported by the “hostname” command; “ALL” means any host.  The idea is that you can maintain the same sudoers file on several hosts.

After the equals sign an optional “run as” list of users can be specified, in parenthesis.  Normally sudo only runs commands as root.  But a user can use “sudo -u username” to run commands as username if that username is in the “run as” list.  Use “(ALL)” to allow a user to run commands as any valid user.

The rest of the line of the entry is the command(s) and their options the user(s) are authorized to run.  Commands should be listed by their absolute pathnames (or as an alias).  You can use the special word sudoedit to allow users to safely edit files normally writable only by root.  (See below.)  You can use ALL to mean any command with any arguments is allowed.  You can use wildcards to specify arguments.

If two or more entries match but conflict in what they allow, the last one wins.

Modern sudo versions are SELinux-aware and include extra features.  Once such feature is the noexec which prevents in many cases a user from gaining a root shell.  For example, if you allow some user to run “vigr” to update the /etc/group file, that user will really be running vi/vim which supports a “:sh” command for a shell prompt.  Since the editor is run as root, so is the shell!

One way to avoid such problems is to use a restricted version of some command.  For example, “rvim” is a restricted version of vim, and “rsh” is a restricted bash.  (rsh also is the name of the “remote shell” command.)  When a command has some sort of shell escape but no restricted mode (that you trust), you should use the noexec feature.

A better way to allow some users the ability to edit normally restricted access files is to add “sudoedit file” to sudoers.  With this setup, the sudo -e file command (same as the sudoedit file command) makes a copy of the file which will be owned by the user.  Next vi/vim is started on that copy.  But unlike normal sudo operation, the command has an EUID of the user, not root.  When the editor exits, if the copy has been modified then sudo will replace the original with the copy.  To allow jdoe to edit /etc/hosts you can add an entry like this:

          jdoe ALL = sudoedit /etc/passwd

(Don’t use /usr/bin/sudoedit; the word sudoedit name is special.)

RBAC (Role Based Access Control)

In conventional UNIX systems, the root user, also referred to as superuser, is all-powerful.  Programs that run as root, or setuid-to-root programs, are all-powerful.  The root user has the ability to read and write to any file, run all programs, and send kill signals to any process.  Effectively, anyone who can become superuser can modify a site's firewall, alter the audit trail, read confidential records, and shut down the entire network.  A setuid program that is hijacked can do anything on the system.  The root user is also anonymous, in that you can’t tell which human executed some command; only “root” is logged.

Role-based access control (RBAC) provides a more secure alternative to the all-or-nothing superuser model.  Using RBAC, one can enforce security policy at a more fine-grained level.  RBAC uses the security principle of least privilege: a user has precisely the amount of privilege that is necessary to perform a job, no more.  Regular users have enough privilege to use their applications, check the status of their jobs, print files, create new files, and so on.  For tasks that traditionally required rootly privilege, a user must assume a role that has sufficient privileges to do the task.  Not every user can assume every role; which users can assume which roles, is up to the administrator.

RBAC works best when the roles are clearly defined in an organization, and don’t change or evolve much or often; and when there are many more users than roles.  This is not always the case, and RBAC is not always the best way to manage access control.  In addition, basic RBAC may grant too much privilege.  (Imagine access to HIPPA-protected medical records: all doctor-roles need access, but only to their patients’ records.)

Because RBAC can be used in cross-enterprise situations, exact meanings of RBAC concepts needed to be standardized.  This was done by ANSI (and INCITS), standard 359-2011, (link is to an older draft PDF).  Work is ongoing to enhance this standard, at NIST and INCITS.

Solaris RBAC

Roles are a special type of user account.  Then run a special version of the shell (there are versions for many different popular shells, see /usr/bin/pf*sh), which allows commands to run that require the extra privilege granted by the role.  Note it isn’t possible to login using one of these role accounts.  Users must first log in as themselves, and then assume the role needed to run privileged commands.

RBAC distributes the various superuser capabilities into rights profiles.  These rights profiles are assigned to special user accounts that are called roles.  A user can then assume a role using the su command, to do a job that requires some of superuser’s capabilities.  Predefined rights profiles (and the root role) are supplied with Solaris.  You create the additional roles you wish and assign the profiles to them.  You can also create new rights profiles.

Rights profiles can provide broad or narrow capabilities.  For example, the System Administrator rights profile enables an account to perform tasks that are not related to security, such as printer management and cron jobs.  The Cron Management rights profile only has privileges to manage at and cron jobs.  When you create roles, the roles can be assigned broad capabilities or narrow capabilities.

A rights profile can include authorizations, directly assigned privileges, commands to be run with security attributes, and other rights profiles.  This should help make it easy to manage the various profiles you may need to define.

RBAC assigns various privileges to roles.  Users are allowed to assume certain roles via su by providing an appropriate password.  Each role has just enough privileges to accomplish a specific task: password reset, DNS administrator, website updater, etc.  In this way, few SAs need the root password to do any task.

RBAC has several components:

A profile is a collection of the Solaris authorizations (i.e., privileges) listed in the file /etc/security/auth_attr.

Profiles are defined in prof_attr.

The actual commands usable in a profile, plus the correct GID and UID to use when running these commands, are defined in exec_attr.  (These are the security attributes discussed above.)  Note only the special “pf” versions of the shells know about such security attributes.  A regular shell will note SUID/SGID bits, but ignore security attributes.

The file user_attr associates user accounts with roles, and roles with profiles.  Although users can be assigned profiles or even individual authorizations, this is not as safe nor easy to administer.

A role is a pseudo account authorized users can use with su to temporarily assume some profile(s).  These role accounts use special versions of the shell (pfsh, pfcsh, pfksh, pfksh93, pfbash, pftcsh, pfzsh, or pfexec), that can apply security attributes.

Example roles include password adm, mail adm, backup adm, login adm, mount, shutdown, ..., and service (start or stop) adm.  (In Solaris 11, root is also a role, not an account.)

You need to restart nscd, the name service cache demon for changes to roles and profiles to take effect.  (“svcadm restart system/name-service-cache”)  Use sm* and role* commands to edit the files.  (Show Solaris 10 RBAC.htm demo.)  See also the profiles command, to see what you can do in a role.

Demo: roles, profiles,  auths|sed 's/,/ /g'|fold -sw 30|sort

RBAC works with a basic privilege set, defined as a system property, and part of Solaris process rights management.  When you specify a set of privileges, you can use the word “basic” to denote this basic set:

# ppriv -l basic
file_link_any
proc_exec
proc_fork
proc_info
proc_session

The way this really works is that the basic set of privileges is really the set of all conceivable privileges an unprivileged user can have.  Sun hasn’t named all those privileges yet (only five so far, for Solaris 10).

You can use ppriv to see what privileges some command needs.  For example:  “ppriv -De cat /etc/shadow” will show something like missing privilege "file_dac_read".  You can then add that privilege to the set for some role or user.

To list all available authorizations, use: getent auth_attr
(To view the authorizations assigned to the current user, use the auths command.)

To list all available privileges, use: man -s 5 privileges
(To list the privileges of roles, users, and processes, use the ppriv command.)

To list all defined rights profiles, use: getent prof_attr
(To view the current user’s profiles, use the profiles command.)

To list all commands with security attributes, use: getent exec_attr

To see a list of all defined roles, examine the /etc/passwd (or similar) and user_attr files.  (To view the roles assigned to the current user, use the roles command.)

(Tip:  If you have the authorizations to run some command, you don’t need to assume a role first.  You can just run the appropriate shell such as pfksh, and from there run the command.)

To create a role and assign a user to it, use commands such as:

# roleadd -K roleauth=user -P "Network Management" \
      netmgt
# usermod -R +netmgt jdoe

A kernel patch Grsecurity is available for Linux that provides RBAC and other security features.  (SELinux also supports RBAC.)

Lecture 13 — PAM (Pluggable Authentication Modules)

PAM Background: [First discussed in Admin II.  Show PAM web resource]

Many interactive commands are security sensitive.  An obvious example is passwd, used to change a user’s password.  Such commands require users to authenticate themselves even though they have successfully logged in to the system.  Also many server daemons carry out tasks on behalf of remote users, and most of these require the daemon to authenticate the remote user.

Early versions of Unix had such programs (applications and daemons) directly read and parse the /etc/passwd file, so they could authenticate users.  This became a problem when the format of /etc/passwd changed to include aging information in the second field.  Every program that needed to authenticate users had to be updated and re-compiled.  Sometime after that single (remote) user databases became common in large organizations, using technology such as NIS.  This meant a second set of such commands (all NIS commands start with the letters “yp” as in yppasswd).  Soon shadow passwords became common, then Kerberos, then different password encryption algorithms (such as MD5), LDAP, etc.  Every such change requires either custom versions of applications and daemons, or a re-write of the existing versions.  And for every program that needed authentication!  (On both Unix and Linux, there are many tools that limit or restrict access to resources available.)

At some point, someone came up with the idea that programs that need authentication should use a standard library for that, which in turn could be configured to use any scheme required by adding new DLLs (or shared object files) which can be invoked by the library.  With this system known as PAM (for Pluggable Authentication Modules) programs would not need to be changed in any way to use different authentication modules.  (Hence the name.)

Modern (and most legacy) applications and daemons that need authentication use PAM.  There are many PAM modules available for every system and new ones can be easily created by programmers.  A nice benefit of this design is that different programs can use different PAM modules for authentication, all on the same system.  In addition, PAM modules can be used for session setup and tear-down, logging, and various similar uses.  You can list for each application a set of modules to try, so if the user can’t authenticate using (say) local files then PAM might try a different database such as Windows AD.  PAM can also be configured to deny access if the resource is busy or based on other criteria such as the time of day.

PAM allows you to provide many restrictions on resources such as programs and devices that a user can access; PAM even allows restrictions such as “only allow 50 simultaneous users of some program (a database for instance)”.

There are multiple implementations of PAM, and a specification (it’s basic).  OpenPAM is a rewrite of the specification only parts of Linux-PAM, by someone who thought Linux-PAM code was of poor quality.  While used with Mac OS and FreeBSD, and others, it has little documentation available and few code contributors, and far fewer available modules than Linux-PAM.  Oracle (Sun) also has their own PAM version for Solaris.  Fundamentally, they all do the same tasks.

Linux PAM Details

PAM is a library that applications needing authentication are compiled with (e.g., login).  For each program that you want to control, add a text file in pam.d with the application’s service name (usually the application’s name).  That file lists the PAM modules that will be used along with any module options.

The man page for a PAM-ified application should say which service name to use.  If not you can guess to use the application’s name as the service name.  (Or read the documentation, or source code, or Google for the answer.)

If an application was compiled with PAM but there is no file in pam.d for it, the pam.d/other entry is used.

(Security: Always have an appropriate other file using pam_deny.so!)

To impose PAM restrictions on a program you create a configuration file for that program in /etc/pam.d/ that calls one or more PAM modules.  Some of these modules can be used to restrict access.  By editing (or creating if missing) these files, an SA implements access policy for the applications and servers on the host.

The available PAM modules are found in /lib/security (however you can add others to that directory, or anywhere).  Some of the more interesting modules are pam_access, pam_console, pam_limits, and pam_time.  Many of these modules have configuration files in /etc/security.  (Show limits.conf.)

The files limits.conf and limits.conf.d/* (and pam_limit) are not used all versions systemd.  With systemd, POSIX per-process resource limits per user are set in /etc/systemd/user.conf.  Setting cgroup (control group) limits (for per-user limits) is also done solely by systemd now, however the means of doing so are not well documented.

PAM modules are typically named pam_XXX.so and are documented in the man pages under pam_XXX (maybe!  Try man pam and/or man pam.conf), /usr/share/doc/pam-version/* and the on-line documentation.

PAM modules can do different things.  Some can do several (usually related) things.  A PAM module may be called within one of four contexts, which determines which one of the four possible tasks you want the module to perform.  Another way to say this is that a PAM module can contain one to four sets of functions (see pam(3)).  Not all modules do anything in a given context (in fact most modules only work in one context).  If some module is missing a set of functions, the module does nothing in that context.

The four contexts (or types) of PAM modules are:

auth modules authenticate users, most commonly from passwords.  But other possibilities include Kerberos tickets, ssh keys, and biometrics.  The standard auth module pam_pwdb (or pam_unix) is used for password authentication and can be configured to say which authentication (password) database to use.  pam_nologin prevents all but root logins if the file /etc/nologin exists (and will show users the contents).  The two functions are pam_authenticate and pam_setcred (used e.g. to get a Kerberos ticket).

On most systems, pam_unix in the “auth” context (and/or the “password” context) is provided the argument nullok, which means an empty password is allowed.  This is generally a bad idea, and should be changed.

account modules check for valid accounts and also control access to resources by other means: time of day, location of user, max number of users, etc.  (I.e., a given account may only be valid at certain times.)  Examples: games only after 5 PM, up to 50 database users at once.  For example root can’t login except at the console.  This is controlled by the pam_securetty module (which lists authorized locations in the file /etc/securetty, which may need adjusting!).  The one function here is pam_acct_mgmt.

session modules don’t control access at all.  Instead they are used to do some automatic setup (once the authentication is successful) and cleanup (when the session ends).  Examples include logging user activity, and mounting (and later un-mounting) a user’s home directory.  The two functions are pam_open_session and pam_close_session.

A useful session module is pam_console, which uses the file /etc/security/console.perms to control access to devices (such as sound devices and removable media drives) and the directory /etc/security/console.apps in which (empty) files can be created with the names of programs that may be run by any user at the console (e.g., halt).

password modules control a user’s ability to update authentication information.  These modules can be used to enforce password changing policies.  The function for this in the PAM module is called pam_chauthtok.

What happens when a PAM-ified program runs

When a user runs an application such as passwd a PAM function (“pam_authenticate()”) is invoked by the command (after running pam_start).  This function locates the PAM configuration file for this application and starts running the modules within, in order, for the auth and account contexts.  (Example: password setting modules are ignored with authenticating a user.)

Each module can either succeed or fail.  The results of the modules called are combined into a single pass / fall result and that is passed back to the application, which should continue on if pass or log an error message and quit if PAM says fail.

When the command terminates it ends the session with a call to the pam_end function.  In between other PAM library functions can be used to check passwords (when changing), and setup and tear-down sessions.

Configuring PAM

These files have the following syntax (described in pam.conf(5) man page):

          context  control-flag  module  options

The file can have blank and comment lines.  Each of the rules should be a single line, but long lines may be split by ending the first line with a backslash.

As mentioned above, the PAM modules listed in the application’s configuration file are run in the order listed, for the appropriate context.  Any module arguments listed for some module are passed to the PAM functions (in that module) that get invoked.

Each module may return one of several different values, some of which indicate success and others failure.  For each listed module, there is a control-flag that controls how PAM will react to a success or failure of individual modules.  When done running all listed modules, the individual module results are combined into a single success or failure result, which is passed back to the application.  While the modern version of PAM allows a complex control-flag field that specifies different actions for different return values, the old style control flags are simpler and are often used instead.

The (old-style) control-flags are: required (the module must succeed.  If a required module fails, the remaining modules are still tried), requisite (a fail-fast version of required), sufficient (if this module succeeds and no previous required modules have failed, no further modules are tried.  A failure of this module won’t affect overall success or failure of PAM), and optional (these don’t affect the overall success or failure of PAM.)  Solaris 10 version of PAM also has binding, similar to sufficient, except that a failure is like a required-failure rather than an optional-failure.

When the type (context) field is preceded with a dash, if that module is missing, the “missing module” message won’t be logged.  The module is still run, however, if present.  This is typically used for modules that may not be present (such as a fingerprint scanner module), and that use a control flag of sufficient or optional.

Two additional control flags in modern pam are include and substack.  These are similar, and cause all the rules of the same type (context) from the named config file to be included.

The options are additional arguments passed to the module’s functions.  Note the unusual quoting convention: if an option includes spaces, surround it with square braces.

Examples

pam_ pwquality is used when changing passwords.  Its arguments determine the password policy.  In /etc/pam.d/passwd put this line:

password required pam_ pwquality.so difok=3 minlen=8 \
 retry=3

The DoD/DISA recommendation is stricter:
password required pam_ pwquality.so retry=3 \
 minlen=14 dcredit=-1 ucredit=-1 ocredit=-1 lcredit=-1

This says: at least 10 characters with at least one digit, upper, lower, and symbol.

(Many people mistakenly think minimum password length policy is set in other files such as /etc/login.defs or /etc/default/useradd, but not so!  Such files are only used under specific circumstances.  For instance, login.defs is only consulted when using shadow password files, not if using LDAP or NIS or Samba.  And even when such files apply, any candidate password must satisfy PAM criteria also.  It makes sense therefore, always to set password policy using only PAM.)

Another policy decision controlled by PAM is whether or not root can log in using the Gnome GUI (gdm).  (Previously, and still for KDM I think, you needed to edit the GDM or KDM config file for this; with KDE that is kdmrc which might be anywhere depending on your system.)  Note on Ubuntu, with no root password set, you couldn’t log in as root anyway — a missing password locks the account.

QU: Suppose you make a copy of /etc/pam.d/chfn in /etc/pam.d/chfn.bak and modified the chfn file so that only root can use the chfn command.  Is this safe?

ANS: Security is a tricky thing.  By putting the backup in the same directory, PAM will apply that policy for a command called “chfn.bak”.  So a malicious user needs only to create a file called “chfn.bak”:

          cp /mnt/flash1/chfn ~/chfn.bak
    ./chfn.bak

In reality the danger is much less than you might think.  For one thing /usr/bin/chfn is not readable (only executable), and thus can’t be copied.  (Of course a user need only use a bootable Cd or flash disk to get root privileges and copy that file.)

For another thing commands have hard-coded in them the name of the PAM configuration file to use; the system doesn’t just use the application name.  Still this is a potential security hole that you should not create.

The ways around this are: create a different directory to hold your backup copies of files, use “RCS” or another VCS to manage backups, or just edit the files carefully, commenting out any modified lines and added additional comments to explain the changes (and allow your normal backup procedures to save older versions).

Not all config files have this problem, but be careful with creating new files in “/etc/*.d/” directories.

Some interesting PAM modules

The pam_stack.so module allowed an administrator to put common PAM policy in a single file.  This module took another config file name as an argument.  When this module ran, it ran all the appropriate modules from the named file.  Modern Linux PAM doesn’t use this module (although older and non-Linux PAM might), and instead has built-in support for this, using include (e.g. “auth include system-auth”) or substack.

Another interesting module is pam_timestamp.so, which records the last time a user has successfully authenticated.  This module can be used to avoid re-authenticating if the user has recently done so.  The sudo command uses this module, recording the timestamps as the modification time of files in /var/run/sudo/username/*.

PAM normally logs the success or failure of each attempt to the secure log.  This can be prevented using pam_succeed_if.so module with the quiet option.

Here’s an example policy for the auth context:

auth   required    pam_env.so
-auth  sufficient  pam_fprintd.so
auth   sufficient  pam_unix.so nullok try_first_pass
auth   requisite   pam_succeed_if.so uid >= 1000 quiet
auth   required    pam_deny.so

(This is a standard part of Fedora’s common policy file, system-auth.)

The first rule sets up the environment (by removing dangerous entries and adding missing required ones; sudo has a similar feature).  This module should pass.

Next is the fingerprint scanner.  If not installed, no log message about a missing module is generated (the leading dash).  If that passes, PAM is done.  If not, keep going.

Next is the standard Unix authentication.  This means standard system functions are used, which in turn are configured by nsswitch.conf.  Usually, that is set to use files (the passwd, group, and shadow files), or some single sign-on database such as Kerberos or LDAP.  If this module passes, PAM is done.

The next line causes PAM to fail quietly (no log message) if the UID is < 1,000 (meaning, a system account).  Regular user accounts have higher UIDs, so this module would pass those.

The last line just fails.  The result is logged.

[Students should examine other policy files, such as the gdm* ones, su, sudo, crond/atd.]

Lecture 14 — Resource Limiting (ulimit, Disk Quotas, cgroups)

It is often the case that you need to limit the use of scarce resources (CPU time, RAM, network connections, open files, etc.)  Such limits can be imposed on a per-process basis, or a per-user basis.

There are a number of different mechanisms used to limit resource use.  Some of these seem to overlap in functionality!

The main tool on Solaris is a zone, which is the major part of a Solaris container.  You can declare resource limits on a container/zone, and restrict users and processes to only using certain containers/zones.  (This is discussed further below.)

Ulimit

The shell command ulimit can be used to enforce POSIX per-process resource limits, on processes started by that shell.  There are both soft and hard limits.  Once set, a user (except root) can lower (down to the current soft limit) but never increase their hard limit.  The soft limit can be decreased, or increased up to the current hard limit.  (root can arbitrartily set these as long as soft <= hard.)  Login scripts can set this to enforce the limits, and there is a PAM module that sets these at login, from a simple to edit config file.  (Show bash; ulimit -a; -Ha; -Hn 1000; -[S]n 900; -Hn 1000; -Sn 1010; -Sn 950.)

If data confidentiality is an issue, you should ensure your programs can’t produce core dumps.  This is done by limiting the size of a core dump to 0 bytes by using the ulimit(1) command before running the program.

(The Solaris command prctl can be used to change these limits on processes, projects, or Solaris zones.  FreeBSD has a similar command, rctl.)

Additional limits can be made system-wide or per-user, using Linux control groups (“cgroups” for short) and/or systemd.  However, how to use these features is poorly documented and not well understood.

Disk Quotas

Disk space used by a given user can be limited using disk quotas (discussed in admin I course).  Quotas are set per user or per group, and apply per filesystem.  Not all filesystem types support quotas, and those that do generally require a special mount option(s) to enable quota enforcement (just “quota” as of ext4).  Demo:

mount -o remount,usrquota
vi fstab # add usrquota, possibly grpquota too
touch /aquota.user # do this on each mount-point
quotacheck -cavuR # initialize quota DB except for /
chmod a+r /*/aquota.user /aquota.user # make readable
restorecon /home/aquota* # Update SELinux labels
telinit 1 # go into single-user mode to check / FS
quotacheck -cvuM /  # initialize quota DB on / FS

(On most systems, you can touch a file to force quotacheck to run on next boot.  For Red Hat systems that file is “/forcequotacheck”.  Then a simple reboot will rebuild the quota DBs safely.)

Linux Control Groups (cgroups)

A powerful but little-understood feature of Linux is cgroups, a way to limit resource consumption by processes.  Often performance problems are caused by some process hogging all the network bandwidth, all the CPU cycles, all the memory, or some other resource(s).  With monitoring, the SA can identify such resource “hogs”.  But it would be better to prevent hogs and report attempts to hog resources.

Some daemons include tunable parameters that can limit what they consume, but many do not.  Instead, you can let the Linux kernel enforce the limits, similar to Solaris containers.  Currently (RHEL6) the kernel can enforce limits on the following subsystems for each task (note the Linux term task essentially means process or thread):

·       blkio — this subsystem sets limits on input/output access to and from block devices such as physical drives (disk, solid state, USB, etc.).

·       cpu — this subsystem uses the scheduler to provide access to the CPU.

·       cpuset — this subsystem assigns individual CPUs (on a multicore system) and memory nodes (on a NUMA system) to tasks in a cgroup.

·       devices — this subsystem allows or denies access to devices.

·       memory — this subsystem sets limits on memory use by tasks in a cgroup, and generates automatic reports on memory resources used by those tasks.

·       net_cls — this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task.  The tc command can then implement queuing rules to limit bandwidth use for tagged packets.

·       freezer — this subsystem suspends or resumes tasks in a cgroup.

·       cpuacct — this subsystem generates automatic reports on CPU resources used by tasks in a cgroup.

·        ns — the namespace subsystem provides a private view of the system to the processes in a cgroup, and is used primarily for OS-level virtualization.  It has no special functions other than to track changes in namespace.

(Only the first six actually control access to resources.)  These subsystems are also called resource controllers, or simply controllers.

Additional subsystem controllers are added from time to time to the kernel.  For example, Linux v3.3 adds a CPU time limit for process groups, and a net_prio to set network traffic priorities by application.  To see the list of available controllers/subsystems for your kernel version, use the lssubsys -a command.

To use these, attach one or more controllers to a hierarchy of cgroups.  Each cgroup has a parent and possibly children cgroups; settings in parents are inherited in children but can be overridden too.  By default, all tasks belong in a default cgroup with no resource controllers attached.  (However, different distros use different defaults.)

Each controller can be attached to a single hierarchy only, but it is common to have two or more controllers attached to one hierarchy such as both memory and CPU.  Once you create the hierarchy (with the one “root” cgroup in it to start) with resources attached that you want to use, the SA next adds additional named cgroups to it for tasks with different limits.

Each cgroup can specify limits for the controllers in its hierarchy.  Lastly, the SA puts one or more processes in each cgroup.  Each process can be a member of multiple cgroups, but only one in a given hierarchy.  By default, any children tasks (remember that a task means process/thread) are in the same cgroup as the parent.

For example, you can attach the memory controller to a hierarchy of two cgroups, one limiting memory to 1GiB and another to 500MiB.  You then put the PID of postgresqld into the 1GiB cgroup, the PID of httpd in the other, and that’s it.  Red Hat systems automatically add daemons to cgroups when they start, by means of a setting in the daemon’s /etc/sysconfig file, or by some default rules in the /etc/cgrules.conf file or by options set in systemd unit files and config files.  (cgrules.conf stopped working after cgroups v2 was released).

The cgroup feature of the kernel is exposed via a “fake” filesystem (similar to /proc or /sys).  You mount these, one for each hierarchy you wish to create. Each mount point defines one hierarchy, with subdirectories defining the cgroups.  Child cgroups inherit some settings from their parent cgroup.  Fake files within the directories are used to configure them and to list the PIDs of tasks.  Only the owner and group members of the files have write permission on these files, and thus control cgroups: change settings, and add/remove processes.

To use cgroups:  [Skip this section; cgroups have been completely rewritten in newer Linux, and have a different user interface.]

# yum install libcgroup # make sure the tools are installed

# chkconfig cgconfig on # A Red Hat convenience

# service cgconfig start  # mounts hierarchies in /cgroup according to defaults

(Take a look now at the hierarchies created.  Inside of each is the default, or root cgroup for the hierarchy, containing the files that can be edited with the limits you want.  For example, in the memory hierarchy, if you want to create a limit for maximum memory usage, it’s stored in a file called memory.max_usage_in_bytes.)

# mkdir /cgroup/hierarchy-name/cgroup-name

# mkdir /cgroup/hierarchy-name/cgroup-name/child-cgroup-name

To specify resource limits you can use the RH convenience utility cgset, or just create the appropriate files with the correct contents.  Each controller has its own set of files.  Child cgroups can set smaller limits than their parent, but not larger.

For example, suppose you have an 8 core CPU and you wish to limit httpd (Apache) to just using cores #0 and #1.  You would mount a cpuset hierarchy, create a cgroup within with that to limit the CPUs used by processes in that cgroup, and add the PIDs of httpd to the cgroup:

# mkdir /cgroup/cpuset
# mount -t cgroup -o cpuset /cgroup/cpuset
# mkdir /cgroup/cpuset/group01
# cd /cgroup/cpuset/group01
# echo '0-1' >cpuset.cpus
# cat /var/run/httpd/httpd.pid >> tasks

There are ways to make all this automatic.  A daemon can be run to automatically put processes into various cgroups, according to the rules in /etc/cgrules.conf.  You can also use the cgclassify command.

Here’s an example to limit Apache to 1GiB of RAM when it runs, using the Red Hat config files method:

First create a group statement in /etc/cgconfig.conf like so:

group http {
   memory {
      memory.limit_in_bytes = 1024M;
   }
}

This will create a cgroup named http in the memory hierarchy, with a 1 GiB limit.  Next, add this line to the /etc/sysconfig/httpd.conf file:

          CGROUP_DAEMON="memory:/http"

That’s it!  Now start the cgconfig service, and then the httpd service.

The definitive document for Linux cgroups (for RH anyway) can be found at docs.redhat.com resource management guide.  See also this much shorter how-to article found in Novell’s (now Attachmate’s) Connection Magazine, Effective Linux Resource Management.

Lecture 15 — Security Policies

The security of a system depends on two assumptions: there is a clear and comprehensive security policy that partitions all the system states into secure states and insecure states.  Secondly, that the security mechanisms prevent the system from entering into any insecure state.  Trust is needed that the security mechanisms will do that.

The heart of security is the security policy and procedures documents.  Some types of policy documents to have include: personal use (or acceptable use policy or AUP, also called the user code of conduct or UCC), email, web server, firewall, laptop, problem reporting policy, and others.  To create a security policy:

·       List the users (a.k.a. groups or communities) for whom you plan to provide services.

·       List the services you plan to provide to each group of users.

·       Describe the security measures to be taken for each permitted access and for each service.  Be detailed and specific!

·       Deny all other access.

·       Provide a procedure to periodically review and possibly update this policy.  Also provide an audit procedure to verify compliance of the actual systems to this policy.  (This audit procedure should say who does the audit, how often it should be done, what should be tested, who violations are reported to, and how violations should be handled.)

A place to start is the current financial statement for the organization.  For each line item with a significant cost or revenue, break it down into tasks and users.  If necessary, break those down further.  Then for each list the users (roles) and services needed by those users to perform their required tasks.

This type of analysis can help with SOX and HIPAA compliance as well.

Other Issues

Note a security policy must address other issues as well, such as data retention, compliance, and procedures for working with law enforcement.

The Electronic Communications Privacy act (ECPA) provides a means for law enforcement to access Internet user information including IP addresses you have accessed.  A law enforcement agency can request an ISP to retain customer information for 90 days and renew its request for another 90 days, providing time to obtain a warrant.

FRCP — The Federal Rules of Civil Procedure include rules for handling of ESI (Electronically Stored Information) when legal action (e.g. lawsuits) is immanent or already underway.  You must suspend normal data destruction activities (such as reusing backup media), possibly make “snapshot” backups of various workstations, log files, and other ESI, classify the ESI as easy or hard to produce, and the cost to produce the hard ESI (which the other party must pay), work out a “discovery” (of evidence) plan, and actually produce the ESI in a legally acceptable manner.  An SA should consult the corporate lawyers well in advance to work out the procedures.

OS Enforced Policies

Some OSes have central policy stores that contain automatically enforceable, organization-wide policies that limit resources and/or process access.  Windows has group policies and Mac OS has an authorization service.  Both of these use a single policy database.  These policies allow specific combinations of (user,program) limited access to specific files and devices, without granting any extra privileges.

There is currently no comparable Unix or Linux subsystem.  SAs usually must use a hodge-podge of different OS-specific and proprietary solutions to define who can do what.  A combination of PAM, ulimit, disk quotas (and restrictive mount options), sudo, group memberships, SELinux rules “sort of” handles this for a single host.  Adding Kerberos and LDAP and a bunch of synchronization tools (such as rsync) can “sort of” handle security policy enforcement organization-wide.  However note sudo grants a user process rootly powers, allowing a badly written (or virus-infected) program to gain root privileges.  Restricting access with SELinux is very difficult and per system.  PAM allows device access if logged into the console (used mostly by RH; Debian systems generally use groups to control access (e.g. cdrom, plugdev, ...).

Issues with pam_console, group memberships, and sudo
[f/ hal.freedesktop.org/docs/PolicyKit/intro-define-problem.html]

·       Mechanisms are coarse grained: (e.g. it’s hard to express “OK to mount removable media but not OK to mount fixed media”.

·       The way pam-console and sudo work is to run (large, buggy) applications (and all their DLLs) as root.  This is in direct violation of the well-known least privilege principle — you want only a small amount of code to run as root, not the whole Gnome desktop.  There is no way to use PAM or sudo to limit the privilege to a small part of an application.

·       Debian based systems usually use group membership to determine if a user is allowed to do something.  (The files are only accessible to those group members.)  But group membership is not a good way to enforce security policy; if a user is a member of a group once, they can always become member of the group again (copy /bin/bash to $HOME; chown to group, set the setgid bit, done).

·       It is difficult for upstream projects (such as GNOME or KDE) to implement features that requires administrative privileges using existing mechanisms, because most OSes have different ways of implementing access control.  As a result such features are handled by OS distributors who each have their own code for doing the same thing e.g. setting the date/timezone etc.  Without such vendor support, there is no clean way (for example) for file sharing applications such as Banshee or Rhythmbox to punch a hole in the firewall.

·       Without a centralized framework, access control configuration is often scattered throughout the system which makes it hard for system administrators to grasp how to configure the system.  There are many different sub-systems, each with configuration files with different formats and semantics.

Red Hat now includes polkit (formerly Policykit) which has a set of tools to manage a policy DB, and an API for applications to use.  (for example, you can use the crconf tool (sourceforge.net) to view and configure the Linux kernel’s crypto API.)

Polkit (previously known as PolicyKit)

Polkit differs from SELinux in several ways: SELinux is static, in the sense that if SELinux (or Solaris RBAC) denies access, only the admin can change that.  In contrast PolicyKit (like PAM) is dynamic, in that if permission is denied, the app can then prompt the user to authenticate at that time).  PolicyKit requires no kernel modifications.  It can use lots of information to make a decision (e.g., when a remote desktop session is active).  PolicyKit is very similar to sudo, except sudo offers checking of command arguments; PolicyKit doesn’t.  However, PolicyKit is designed to run very small, focused executables as root, not the whole application as with sudo.

Applications that use PolicyKit are split into two parts; each runs as a separate process.  Only the part that needs the extra access runs as SetUID (using the sudo-like program pkexec).  The other part runs as the current user.  Note that either or both parts may use PAM as well.

To (say) disable networking, the unprivileged part sends a message to the privileged part (in this case, DeviceKit).  This part uses the Polikit DLL (libpolkit) to see if the user is allowed the requested action.  That library consults XML policy files, kept in /usr/share/polkit-1/actions.  For network control, the file used on Fedora is org.freedesktop.NetworkManager.policy.  (Show.)

Polkit will return one of: yes (the user is allowed to do the action), no (access denied), user authentication required (the user must supply their own password, like running passwd or chfn requires), or admin authentication required (the user must supply the root password).  If the action is allowed, the SUID program pkexec runs it.

Also, if user or admin authentication is used, the result can be cached for the life of the process, the duration of the user’s login session, or permanently.  (Not sure if this is true; with Fedora, the man page only says “caches for a short while”.)  This means if the process needs to repeat the privileged action, the user doesn’t need to keep authenticating.

By editing (or adding policy files), an SA can change the policy.  For additional details, see polkit(8), or
 hal.freedesktop.org/docs/PolicyKit/model-theory-of-operation.html.

Each file is named for the actions it has control over.  These policy files define the default policy and are installed automatically when the application is installed.  The configuration file format, along with examples, is described in the man pages for Polkit(8) and pkexec(1).

Secure Boot

This is a combination of hardware and firmware, designed to prevent untrusted and unauthorized OS code (rootkits).  In practice, this seems more like a Microsoft - Intel collusion to prevent users from replacing Windows with other operating systems.  This is because there doesn’t seem to be any provision to authorize any other kernel code (such as WebOS or Linux).  Apple apparently has similar plans.  MS has announced Windows 8 will require Secure Boot hardware.

Lecture 16 — Incident Response

An incident is any reported or discovered problem.  Most incidents get reported by customers/users via the help desk and/or trouble-ticketing system.  However you can purchase entire “incident management systems”, which I think are just glorified trouble-ticketing systems bundled with a help desk.

Most incidents are not security related and are handled in the usual way (assign the issue to some SA or developer).  However security related incidents require immediate action.

A security incident is any unlawful, unauthorized, or abusive action that involves a computer or network.  Examples of such incidents include theft of IP, spam, harassment, unauthorized use of systems (theft of service), embezzlement, denial of service, and extortion.  An incident response may be appropriate for non-computer security incidents, when digital evidence may be expected to be obtained.

Once a security incident has been identified you (or someone) must react to it.  Most large organizations define a group of individuals with various expertise as part of a computer security incident response team (CSIRT).  This group includes legal, technical, financial, and possibly other experts (e.g. political).  It is dynamically formed when needed from the organization’s employees and consultants.  This is useful because proper handling of evidence, detailed knowledge of organization policies, and a working relationship with law enforcement, ISPs, and others is difficult without special (and regular) training.  CERT, SAN, and others now offer courses and certification for incident response.

An organization’s incident response policy is determined by the answers to the question “Is prosecution possible?”  For a home or SOHO user, the answer is likely “never”.  In that case, your response need not worry about collecting and preserving evidence, and instead focus on stopping the attack and resuming operations as quickly as possible.

If prosecution is posible then you need to collect and preserve digital forensic evidence, in a legally acceptable mannar.

Once an incident is suspected the first step is to confirm something has happened.  This includes an initial investigation and recording what is found, including the details that led to the investigation.  Once confirmed the CSIRT should be formed.

Next, notifications may need to be sent ASAP for legal reasons to affected stakeholders.  (For this reason there is often a lot of time pressure on the CSIRT to hurry the investigation.)

Based on what is known at this point, policy should determine what action or actions to take, and in what order and when to take them.  The range of possible responses should be considered, including civil, criminal, administrative, or other actions.  The CSIRT must also decide to “pull the plug” to prevent any further damage, or to investigate further before taking action.

Pulling the plug may have consequences worse than letting the action continue, and also risks losing RAM-based evidence.  If you power-down the machine, you can then make images of the hard disks for evidence.  If you wait to power down, and try to run commands to preserve evidence, those commands will alter the state of the machine and may not work at all (if a root-kit was installed).

A more thorough investigation should now be conducted, preserving any collected data’s evidentiary value (if legal action is even remotely possible).  The result of the investigations should be reported to upper management, who must make the final decisions as to a course of action.  Note in some cases the SA has legal or ethical requirements on reporting incidents, regardless of the wishes of management.  Sometimes the report will result in a request for a more thorough investigation.

In the end you resolve the incident, usually by cleaning up the damage, plugging the holes used, getting back in operation, filing final reports, and instigating educational initiatives to prevent similar problems from occurring in the future.

Some possible incident responses include:

·       Do nothing

·       Discourage, limit, or block access to offending users (rate limiting in iptables, PAM)

·       Prevent exploit (update firewall, change password reset procedures, ...)

·       Stop attack ASAP

·       Let attack continue and trace back to originator

·       Gather evidence (forensic) for possible use later RAM snapshots, network traces)

·       Report attack to originating site (e.g., abuse@domain.com, or use whois to determine email/phone number to contact)

·       Report to various public advocacy/service organizations (e.g., Vilpul’s Razor, DCC, DNSBL, your ISP, their ISP, ...)

·       Report to various interested parties and decision makers within your organization

·       Report to affected customers/users

·       Take legal action: criminal (FBI, local law enforcement, NSA, DHS, SEC, etc.) and/or civil (report to lawyer, insurance agent, ISP, police)

A Step by Step Guide for Incident Response

The first rule to handle a security incident is: Don’t panic!  The security policies should specify what must be done.  In general:

1.    Verify there is a problem.

2.    Stop any attack in progress (unless you want to attempt to trace the intruder).

3.    Restore/recover the corrupted systems.  In the process a lot of forensic data will be lost; some can be saved quickly enough that it is worth the time, however most of the time you need to get the systems back on-line rapidly, to prevent lost revenues and damaged reputations.  (Note how a backup system is useful: you merely need to switch to that, and then you can take your time with the compromised system.  Typically, you power it off and use forensic tools to image the disks.).

If your system is restarted, the attack may continue.  You need to prevent that before restarting.

4.    Determine what the intruder did and how it was done.  This will show a vulnerability in your system that needs to be addressed quickly, or the intruder will just break in again.  If the vulnerability will take a lot of time to patch, usually a temporary measure can be taken (added a new firewall rule to disable just the one vulnerable service).  This step can take hours to months of time!  At the beginning, and throughout this effort, you need to make incident update reports.

Most attacks today are done by script-kiddies, who leave traces of their activities in the log files and elsewhere.  Serious attackers won’t leave such traces for a novice investigator to find, but it can’t hurt to look before wiping the disk or throwing it out.

Boot using a CD-ROM live distro and use its tools to examine your password and group files, log files, etc.  Check the MD5/SHA sums for commands such as ls, ps, who, etc., with known good values from a similar system.  Check the reported sizes of disks and filesystems, and look for gaps or hidden disk areas.  Look for cron and at jobs that don’t belong, modified boot, service, and login scripts, etc.  Use pwck on the suspect passwd and shadow files.  This should show any bad entries, however if your system was hacked the user name showing in lsof or ps may be faked.

Try to figure out how the intruder got into your system.  Check the network logs to determine when the attack started, and then examine the relevant host log file entries to see what happened at that time.  Look for weak passwords with some tool such as John the Ripper.

Are you running insecure versions of software?  Do you have insecure configurations of servers such as permitting unrestricted uploads via FTP, WebDAV, etc.?  Have you enabled security features, removed unneeded software, used firewalls, and removed old user accounts?

5.    Report the incident.  (Start at same time as step 4.)  This step depends on your policies.  Usually the incident needs to be reported to someone but depending on the policy (and pertinent laws) you may need to report internally only, to law enforcement agencies (police, FBI, NSA, DHS, SEC, FCC, CERT, ...), to your insurance company, to your ISP, and to affected customers and/or users.

Depending on whom the report is for it will include more or less detail.  Technical details ranging from router and switch logs to system logs and even whole disk images may be included.  The report should include:

·       when you discovered the problem (or when it was reported) and its duration (to help focus attention on the relevant portion of the logs)

·       the steps you took already (and are currently taking now)

·       the projected impact (a dollar amount may be needed for insurance purposes)

·       the steps you plan to take in the future

·       any other relevant information (say about past incidents)

Note that any saved log files, directory listings, backups, and all other forensic evidence you collect may be used in a court proceeding someday.  There are very specific rules for handling and storing such evidence.  Failure to follow these procedures breaks what is called the chain of evidence and negates most or all of its value as evidence.

Read the on-line resources for incident response for more detailed information.

Lecture 17 — Using Crypto tools: Checksums, CRCs, Digests / Hashes, and MACs

(The definitions are repeated here from “Cryptography Overview” on p. 77.)

A checksum (or just sum) is a simple way to check for errors.  The bytes (or words) of some file are added together, ignoring any overflow.  The result is a checksum.  This was used by the Roman bishop Hippolytos as early as the third century.  Today money, credit card numbers, ISBN numbers, etc. all use checksums. 

A CRC (cyclic redundancy check) is an improved version of a checksum: the data is considered a very large single number, which is divided by a special value.  The remainder becomes the CRC.  Unlike a checksum, a CRC will detect transposed values and many other types of problems that would get past a checksum.    CRCs are used in networking.  (Demo cksum, which uses the Ethernet CRC defined in ISO/IEC 8802-3:1996.)

Neither of these methods is useful for security applications.  It is not difficult for an attacker to cause a modified file to have the same checksum or CRC as the original.  Instead checksums and CRCs are designed to preserve message/file integrity.

A [secure] [message] digest and (a.k.a. MD) is usually used instead of checksum or CRC for security applications.  Also called a fingerprint, hash, [secure] hash [code] (SHA), file signature, or message integrity code (MIC).  These have the same overall purpose as a checksum or CRC but the algorithms used makes it very difficult for an attacker to predict what changes are needed to cause a particular result.  So unless the attacker can change the MD for a file, you will know if the file was modified.

A hash function is any well-defined procedure or mathematical function, which converts a possibly large amount of data (a message) into a small datum (the hash, hash value, or hash code).  The hash is usually a single integer with the property that a change to the message will change the hash.  A hash is often used to speed database lookups, but some are not secure enough (that is, hard to generate a specific one) to use for MDs.

A cryptographic hash is a one-way function that supposedly can’t be easily reversed (i.e., you can’t find the original message just given the hash), faked (i.e., you can’t easily change the message without also changing the hash), or spoofed (i.e., you can’t easily find a second message with the same hash as another message).  Some better-known cryptographic hash functions include MD5 and SHA-1.

Many people use the terms checksum, digest, hash, etc. interchangeably.  But you shouldn’t!

As mentioned previously, MACs (message authentication code) differ from MDs, in that MACs also use a shared key to compute the digest; both the sender and receiver must share this key to generate (sender) and validate (receiver) the MAC.

To check downloaded files weren’t corrupted, you use md5sum to generate a new hash and compare it with the downloaded one (usually available on the web site where the file was downloaded).  (Other tools such as sum also generate hashes but are not secure nor are as commonly used today as MD5.)  Usage:

md5sum file  (md5sum -c|--check file.md5)   Demo on YborStudent:

   wget http://hostopia.samba.org/samba/ftp/old-versions/samba-2.2.5.tar.gz
   wget http://hostopia.samba.org/samba/ftp/old-versions/samba-2.2.5.tar.gz.md5

   md5sum -c samba-2.2.5.tar.gz.md5
   echo 'oops' >> samba-2.2.5.tar.gz; md5sum ...

sha1sum works the same.  See also gpg --print-md md5 file.

Use gpg --version to see a list of other supported “Hash” algorithms.  Use gpg --print-md algorithm msg to print the hash, and --print-mds to print a hash for all available algorithms.  Note: Using GPG for hashing is a good idea, since it is available for all platforms worldwide, and you can count on repeat­able results.  (Some folks prefer to see such errors.)  See also the command openssl dgst algo [file...]  (see man dgst).

digest on Solaris is another tool that produces digests with a choice of algorithms. 

Comparing the Strength of Secure Hashing Algorithms

The security strength of a hashing method can be expressed as the number of possible passwords an opponent would need to process to guess the original message.  (Of course, the attacker could get lucky on the first try but the odds of winning the lotto every week for years in a row by buying a single ticket each time are better.)

Generally, the number is 2 to the power of the length of the hash.  However some attacks take only 2 ^ (length/2) for many algorithms in use today.  The security number is the exponent used here, generally (hash length)/2.  So the security of MD5 is 64, of SHA-1 is 80.  (I think you can calculate the strength of DES crypt the same way, so that would be 32.)

Today (2006) MD5 or SHA1 CRCs are often used.  However, recent analysis of MD5 has shown that in a few specific applications it is weaker than originally thought.  (It is still much stronger than the original crypt!)  SHA-1 (a patch of the original SHA to address a weakness) suffers from the same flaw, reducing its strength to 69.  Note this doesn’t affect all uses of MD5, such as with HMAC (RFC-2104) which in turn is used with IPSec.

Newer versions of SHA that result in longer hashes are available: SHA-256, SHA-384, and SHA-512.  The NIST is phasing out acceptance of MD5 and SHA-1 by 2010; see U.S. Gov. standard FIPS-180-2.  For approved algorithms, see secure hashing at csrc.nist.gov/groups/ST/toolkit/.

Digital Signatures

A digital signature is the most secure way to verify a file wasn’t corrupted, especially if you can get the matching public key from a different source.  GPG (previously PGP) is used for this.

Digital signatures that use PKI (public keys) are as secure as HMACs but don’t require a shared key, and are thus much more convenient.  As long as the attacker can’t guess the private key, they can’t make a valid signature.

Any hacker who can upload a corrupted file to an FTP site can also upload the matching MD5 file.  But only the real poster has the private key needed to produce a digital signature for some file.  To validate some file that has been signed, you need the public key.

Two commonly used standards for email digital signatures are OpenPGP (implemented by gpg or the older pgp) and S/MIME.  S/MIME uses PKI certificates, and is mostly used to sign email, not encrypt it.  (It can be used for either or both functions, however two certificates are needed to both sign and encrypt.)  GnuPG is popular since it doesn’t require any certificates.

To validate the Linux kernel source, you must get the kernel.org public key and import it into your PGP or GPG database of keys (called your public keyring).  Here’s how:

GnuPG (or GPG)

GnuPG (Gnu Privacy Guard, more commonly referred to by the command name of gpg) is the modern version of PGP (Pretty Good Privacy).  Both GPG and PGP were written by the same person, Phil Zimmerman.  GPG implements the OpenPGP standard, RFC-2440.  The gpg command supports over 70 sub-commands, each with options.  More information (including the latest software for download, and documentation) are available from www.gnupg.org.

You must initialize gpg by running it once, to create the directories and files (empty keystores) it uses before attempting to verify any files.  The first time you run gpg it will create a ~/.gnupg directory and some files.  Next, obtain the public keys you want.  The easy way is from a keyserver (see my website for a list of keyservers).

Using gpg part 1: verify a signed downloaded file

If some file was digitally signed, you can verify the file’s integrity using:

   gpg --verify file.sig

The file and the signature for it (in file.sig) need to be present.  This operation requires the matching public key for that signature is in your gpg keystore (the database of public keys), which is often called a keyring.  If missing, gpg will tell you the KeyID it needs.

You can import someone’s public key with either of the following commands (use the first if you’ve downloaded (or received as email) a key, use the second to search on on-line directory/keystore):

 gpg --import keyfile
 gpg --keyserver wwwkeys.pgp.net --recv-keys KeyID

Here’s how to get the key for the Linux kernel source:

gpg --keyserver wwwkeys.pgp.net --recv-keys 0x517D0F0E

If you don’t know the KeyID number, you can search the on-line directories using email addresses or DNS domain names.  (See my home page (scroll down to the “Privacy” section) for a list of such key servers.)

RPM and yum can use digital signatures to verify packages before installing, but again you will need to import the public keys for those repositories (and not onto your personal keyring either).

To obtain the signature for the kernel source code, use your web browser and go to: www.kernel.org/signature.html.  Scroll down until you see “The current Linux Kernel Archives OpenPGP key is:”.  Copy what follows starting from the “pub 1024D/...” line up to and including the

-----END PGP PUBLIC KEY BLOCK-----

 line into a file, say ~/kernel.pubkey.  Then add this key to your GPG DB with:

   gpg --import kernel.pubkey

Now you can verify the signature for any file signed with the matching private key.  Download both the file and the signature file, then verify like this:

wget ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.19.tar.gz
wget ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.19.tar.gz.sign
gpg --verify linux-2.4.19.tar.gz.sign  linux-2.4.19.tar.gz

(Project: create own keys, send email with digital signature and encryption.)

The Issue of Trust

Receiving a key is dangerous since you don’t really know if that is the key of the person or organization you think it is.  With gpg, each person’s public keys are digitally signed by others.  If you trust any of the others who signed the key, you can have confidence that the key really belongs to the person who claims it is theirs.

Gpg will tell you when you verify a signature that the signature is good (or bad), and who it belongs to, and then it will tell you how much to trust that information.  If the key was signed by someone you trust fully, or signed by three people you trust a little, then you can have high confidence the key belongs to the person who claims it is theirs.  Gpg requires you to decide how much you trust a key you are importing.

Suppose you import key “A” from user “Al”, and trust that this is really Al’s key.  Furthermore, you know Al personally and know that if Al vouches for someone’s key, it is very likely to be that person’s key. Then tomorrow you need to verify a file signed by user “Bob” with key “B”.  If key “B” was signed with Al’s key, you can trust that key “B” really belongs to Bob.

Using GnuPG part 2

Generate a new pair of keys using:      gpg --gen-key

Use DSA and Elgamal (default) as the encryption type.  Use key size of 1024 or bigger, the default=2048.  (Bigger than this doesn’t improve security much.)

Always use an expiration date!  For your first key use a short period, a couple of months.  For a “production” key use something like 5–10 years. 

Next enter your full name, your email address (associated with this key) and a comment (may be blank, or may be a title, a nickname, etc.).

Next, enter a password (since blanks are allowed, it is called a passphrase).  The private key is stored in a file on your computer.  If you don’t encrypt this file with a password, then it is easier for someone (or virus) to steal your private key!  Note if you have multiple private keys, then each will have its own password.

While it is possible to not enter a password (thus making use of the key easier later), this is not recommended.  Instead, you should use agent software.  gpg-agent is available in gpg since version 1.9+.  Your system also comes with ssh-agent, which works similarly but for ssh keys only.  Later we will show how to use that.

The idea of agent software is that at login time, this loads up with all your private keys.  It holds the passwords of each, in a master file, protected with its own password.  At login time, you start the agent, providing it with the master password.  All future uses of encryption will ask the agent, and not interactively ask you, for the private key.  This means you don’t have to enter passwords every time!  The agent stores keys only in RAM and never on the disk.

The commands for listing and exporting keys are:

gpg --list-keys
gpg --list-secret-keys
gpg --export [KeyID]

You can upload your key to a keyserver with:

          gpg --keyserver wwwkeys.pgp.net --send-keys KeyID

You can edit keys, including the level of trust you have that this key really belongs to this user id, and also sign keys (proclaiming to the world that you certify this key as belonging to this user id) with:            gpg --edit-key KeyID

This brings up an interactive mode, enter “help” to see a list of commands.  For example, use the command “trust” to change the level of trust you have in a key.

To encrypt some file for a user, you need their public key:

  gpg -a --encrypt -r KeyID > file # demo with 1184601A

The “-a” says to use ASCII output format.  Without the “-r KeyID” argument, you will be asked to select a key to use.

To decrypt something that was sent to you, you need to use your private key:

gpg --decrypt file  # will ask for your passphrase

To sign a file with your private key, use:

gpg -a --output file.signed --sign file

This command encrypts file with your private key and saves the result as file.asc by default.

This isn’t very useful, as you must decrypt the file to see the original document.  A more useful way of signing the document is to just sign a message digest, and append that to the file.  This is commonly done for email and similar text files.  To sign the document while keeping the original unencrypted, use:

   gpg -a --output file.signed --clearsign file

For files you plan to make available on the Internet it is usually most convenient to put the signature (the message digest encrypted with your private key) in a separate file, usually with the extension “.sig”.  This is done as follows:

   gpg -a --output file.sig --detach-sig file

To verify a signed file use:        gpg --verify file.signed

To verify a file signed using a detached signature, you need both the file and the signature file:

   gpg --verify file.sig
   echo 'opps' >> file
   gpg --verify file.sig

Some other fun gpg tricks:

To encode/decode a binary file (for email say), gpg can be an alternative to uuencode or MIME:

gpg --enarmor < binary-file > text-file

gpg --dearmor < text-file   > binary-file

To generate checksums you can use the command sum, md5sum, sha1sum, or digest (if your system has these commands).  But they may not work identically across all systems (they should but often don’t).  Gpg supports a large range of checksums (a.k.a. hash, a.k.a. message digest) with the “--print-md type” option, where type is the algorithm to use for the message digest.  You can run “gpg --version” to see a list of supported message digest algorithms (look for the “Hash:” line):  gpg --print-md md5 file

Gpg also can use symmetric (shared or private) key encryption methods, used to protect a file with a password.  The default cipher used is CAST5 but another supported cipher may be chosen by using the --cipher-algo option.  The list of supported ciphers is shown in the output of “gpg --version”.

    gpg -c file  # or --symmetric

This prompts for a password, and creates file.gpg.  To decrypt, just run “gpg file.gpg”.

Run this demo on YborStudent (to show xor):

# Create a text file:
echo this is a secret > msg

# Show various checksums:

md5sum msg > msg.md5sum

sha1sum msg > msg.sha1sum

sum msg

sha1sum -c msg.sha1sum

md5sum -c msg.md5sum

gpg --version

gpg --print-mds msg  # or --print-md md5 (or sha1, ...)

# Encryption/Decryption demo:

xor2 < msg > msg.xor  # password "foo"

rm msg;  od -bc msg.xor

xor2 < msg.xor > msg

cat msg; less bin/src/xor.c  # easier to read than xor2.c

gpg -c msg  # password "foo"; creates msg.gpg

rm msg;  gpg msg.gpg;  cat msg

Many secure email clients today use S/MIME or PGP/GnuPG.  However they don’t use that correctly and a flaw (named EFAIL) has be discovered (5/2018) that could allow others to decrypt your emails.  Until the mail clients have been patched (could take awhile), you should disable such plug-ins and use a separate program to encrypt/decrypt emails.  (Thunderbird with Enigmail has been patched.)

Lecture 18 — SSH Security Features and Advanced Use

[Basic use of SSH, including key generation, is described in the Networking class (Remote Access).]

SSH (Secure SHell) is a now common protocol used for remote access (replacing rlogin, telnet), file transfer (replacing ftp), and remote command execution (replacing rsh).  It accomplishes this using encrypted communications packets which provides great security.  SSH can also use public-key encryption techniques to authenticate users, avoiding the use of weak passwords (and incidentally automating the login process, so you don’t have to type a password!)  The use of keys means passwords are never sent across the network; not even your keys are sent (a challenge-response system is used).

Security is provided through a Diffie-Hellman key agreement (similar to the HTTPS protocol).  (Each host has a host-specific key, normally 2048 bits, used to identify the host.)  This key agreement results in a shared session key.  The rest of the session is encrypted using a symmetric cipher, currently 128-bit AES, Blowfish, 3DES, CAST128, Arcfour, 192-bit AES, or 256-bit AES.  The client selects the encryption algorithm to use from those offered by the server.  Additionally, session integrity is provided through a cryptographic message authentication code (hmac-sha1 or hmac-md5).

In addition SSH can be used as a secure tunnel to create an application layer point-to-point VPN (Virtual Private Network).  This is referred to as port forwarding.  SSH’s original protocol turned out to have some weaknesses, so a second version was created which is referred to as SSH2.  There are a number of SSH packages available for all platforms.  One of the best is OpenSSH (www.openssh.org/).  This package includes the server sshd and a number of client programs: ssh, sftp, scp, and several other utilities.

User Setup of SSH

User configuration is in ~/.ssh/*.  This directory should be chmod 700.  To generate the keys you need use the commands:

ssh-keygen -t ecdsa -C 'auser@wpserver'
cd ~/.ssh; cat id_dsa.pub >> authorized_keys

Without the “-t type” option, ssh-keygen generates RSA keys only usable by SSHv2.  See the man page for the different key types available; generally “rsa” or “dsa” (both for SSHv2) are used; if available, ecdsa uses ECC keys.

The keys are only used for authentication in SSH; after that, a session key is generated and symmetric key encryption is used for the session data (blowfish, idea, AES, etc.)

The keys go into files id_rsa, id_dsa, or id_ecdsa, and <the same>.pub (SSHv2), or identity and identity.pub.  (SSHv1).  The .pub file should be appended to authorized_keys on every system you want to have password-less access to; that includes the current host if you wanted to play around with “ssh localhost”.

Note that the man page says the “-C comment” option is only supported for SSHv1 type RSA keys, but it works for me on SSHv2 RSA or DSA keys.

The private keys can be encrypted (with 128-bit AES currently), which means you’ll need to enter a password (passphrase) each time you use them.  If you don’t encrypt your private keys, you are taking a large risk.  However if you do encrypt it, you will need to supply the password each time in order to use it.  You can use an agent to cache the decrypted key in RAM (the agent process) and set some environment variables so agent-aware utilities can get the key from the agent.  Then you only have to supply the password once per session.  (The keychain agent caches these in such a way that they remain in RAM until a reboot, even across login sessions.)  Agents are discussed below.

Using SSH

(For Windows, See PuTTY.)  When executed without further qualification:

ssh hostname

or       ssh user@hostname  # or: ssh -l user hostname

establishes a secure shell (login) session with the named host.  It finds the private key (id_dsa, id_rsa, or identity) on the local system in the current user’s ~/.ssh directory, and looks for the matching public key within the authorized_keys file on the remote system.

You can also use ssh to run a command remotely; any I/O redirection to/from ssh will have that data redirected to/from the remote command:

   echo hello |ssh user@host 'cat >~/hello'

or: ssh user@host 'cat >>~/.ssh/authorized_keys' <id_dsa.pub

Once the connection is made, you can communicate with the local ssh process (rather than the remote shell) using various escape sequences (ex: \n~?).  See the ssh man page for details.

To allow password-less access to host foo.com you must append the public keys from any system you access directly (such as your home/work computer) to the authorized_keys files on foo.com.  An easy way to do this (with OpenSSH) is to use “ssh-copy-id -i id_dsa [user@]host”.

Some systems have different file formats for keys than OpenSSH, so you need to convert them to the proper format before adding them to your authorized_keys file.  Most public SSH keys are in RFC-4716 format, and can be converted to OpenSSH format this way:

   ssh-keygen -if some-key.pub >> authorized_keys

(Several other formats are supported as well; see the “-m” option.)

OpenSSH Remote Login

Once login succeeds, SSH sets up a basic environment (try “ssh localhost env”).  It then will add the environment in the ~/.ssh/environment file to that (this is controlled by the server configuration, as is everything else.)

Ssh then looks for ~/.ssh/rc and runs it if found.  If not, it looks for /etc/ssh/sshrc and runs that.  After that, the users shell is started (and normal startup scripts are then run).

It is possible to have multiple private keys, even on a single system.  This would be useful when you have multiple identities (like a super-hero), or when you have special keys for specific uses.  You must use a command-line option to tell ssh which file to use (for your private key) rather than the default (so name them well).

The authorized_keys file

Another issue is the ability to restrict SSH port forwarding, or to prohibit shell access.  If some user account exists for a limited purpose (such as backups only), you need to be able to enforce those limitations.  Port forwarding is used to create a tunnel from a local client to a remote service; the ssh program will forward packets through the tunnel based on the port number.  (Note the user account must exist on the server end to make a tunnel to that server.)  But on the server end, you may not want that user account used for any but a tunnel to one specific service.  Port forwarding can be restricted by options added to the authorized_keys file.

The authorized_keys file can contain blank lines and comment lines (lines starting with “#”).  Other lines contain a single key each, in this format (fields separated by spaces):

options key-type key comments

(Older SSH1 format RSA keys don’t contain a key-type and instead have other fields, but still have options at the front and comments at the end.)

The sshd man page lists the options you may include here.  The options (if present) consist of comma-separated name or name="value" pairs.  No spaces are permitted except within double quotes, and the name is case-insensitive.  There are many options available; some of the more useful include:

from="pattern"  The use of this key is only valid from the FQDN of the listed hosts.  The pattern is a comma separated list and can include shell wildcards “*”, “?”, and not (“!”), or IP addresses.

command="command"  When this key is used successfully then only run this command.  (The command supplied by the user, if any, is ignored, and no shell session is started.)

Other options include: environment="NAME=value", no-port-forwarding, and no-X11-forwarding.

Here is a sample configuration for the SSH authorized_keys file on a production server, for the user backup when using the backup program rdiff-backup:

command="/usr/bin/python /usr/bin/rdiff-backup⤸
 --server",no-agent-forwarding,no-port-⤸
forwading,no-user-rc,no-X11-fording,no-pty rest-⤸
of-key...

The sudo config in this case requires the backup user to be able to run rdiff-backup as root:

rdiff ALL = (root) NOPASSWD: /usr/bin/rdiff-backup

Using ssh-agent

From your physically secure computer in your private (and kept locked) office, it may be safe enough to not password protect your private key.  But more likely not!  Using keys is done to provide extra security, not to make remote access more convenient by allowing users to login without passwords.

Still it is a pain to type in a password when it is so easy not to encrypt your key.  How tempting!  To make using keys easier without storing them in unencrypted files, you can setup ssh-agent, which holds your private keys in RAM and supplies them to the ssh (or other programs) as needed.  The programs communicate with ssh-agent via sockets or named pipes; the path to those (and other required information) are put into environment variables.  This implies that you must start ssh-agent first, and have it in turn start your shell or X session.  You can also source the output ssh-agent from a login script; this is most common.

Run ssh-agent bash & for a temporary session you can play with, without modifying login scripts (and logging in again).  In a login script you can use the “-s” option to generate the proper setup commands.  Use it this way in your login script (demo: ssh-agent -s, to show the environment variables created that allow agent-aware utilities to communicate with the agent):

eval `ssh-agent -s`

To stop ssh-agent use the “-k” option.  (This command can be put in your ~/.bash_logout file.  Gnome and KDE can use similar files, called when the GUI session is ended.  CDE uses ~/.dt/sessions/sessionexit.)

Initially ssh-agent doesn’t hold any keys.  You use the ssh-add file ... command to add the private keys from file... to ssh-agent (“-l” and “-L” lists the keys currently held).  ssh-agent can also extract private keys from a smart-card reader so you don’t have to swipe your smart card every time!

A better key agent is keychain, which is actually a Bourne (!) shell script that starts ssh-agent and/or gpg-agent as needed, loads in the specified keys, and stores the environment variable settings needed so they can be read when you next login.  The agent(s) aren’t stopped after you log out.  Add this to your login script:

eval `keychain --quiet --eval id_dsa`

    (See Configuration below for a discussion of ForwardAgent setting.)

Running Commands Remotely

Commands can be executed remotely by adding them to the command line.  For example:
    tar cvf - some_dir| ssh user@host "dd of=/dev/tape"

This example uses SSH to backup files from the local system to a remote tape drive.  Try running other simple commands such as who or uptime.

This example runs a command remotely, reading from a local file (saved remotely), and saving the results locally:

  ssh user@host \
   'cat > file && big_app file && rm file' \
   <file >file.out && cat file.out'

X-window commands can also be run through an SSH tunnel.  If you are running an X server locally, any X-client can be run on a remote host and have its GUI appear on your local system (which also sends events such as mouse clicks and typing back to the remote application), without SSH.  However this is dangerous as X security is very rudimentary (man xhost) and the session itself is not encrypted.  Instead you can run the remote GUI application and have it send its X input/output via an SSH tunnel.  For example:

ssh -nX YborStudent.hccfl.edu "gvim foo" &

or:      ssh -X hostname xtop (or xterm)

These commands (options in ~/.ssh/config permitting) will set DISPLAY in the environment of the command on the remote host, to cause it to open windows on the local host via the SSH tunnel.  (The “-n” closes stdin, needed when running ssh in the background; this is mainly useful when running remote GUI apps.)  Of course X window must be running locally for this to work!  By default, OpenSSH may not allow X forwarding; see /etc/ssh/* to check.  To override the default (if not permitted), use:

echo 'ForwardX11 yes' >>~/.ssh/config

Port Forwarding

Suppose you (at host CLIENT) want to access some service (some daemon-port at host SERVER listening on a well-known port).  Sadly, you find a firewall between CLIENT and SERVER prevents access to that service’s port number.  If SSH is allowed, you can create a tunnel between CLIENT and SERVER.  This is sometimes called an application-layer (or L3) point-to-point VPN, although OpenSSH does support true VPNs too (see below).

Securely forwarding packets received on some port at CLIENT, and delivering them to some service listening in on some port at SERVER can be useful.  For example, your mail reader on CLIENT thinks it is talking to some mail server listening at CLIENT:client-port.  The mail server (on SERVER) thinks it is receiving a packet from a MUA at SERVER:server-port.  ssh on CLIENT acts as a proxy server, listening for packets sent to client-port.  It then forwards those packets via the SSH tunnel to sshd on the SERVER, which acts as a proxy client, forwarding packets to whatever is listening at server-port.

Sshd need not be running on SERVER!  It can be on some MIDDLE system that (a) CLIENT can reach with ssh, and (b) can send packets to SERVER.  (I.e., the MIDDLE system must be across the firewall that is blocking direct access.)

Most commonly, MIDDLE and SERVER are the same host.  Then sshd can efficiently use a Unix socket to talk to the server daemon.  Note that when sshd on MIDDLE connects to SERVER, it will resolve DNS names using its resolver, not the one on CLIENT.  (So using “localhost” for the SERVER refers to MIDDLE and not CLIENT.)

To create a (application-layer) point-to-point VPN, use the -L and -R options to forward a specific port on CLIENT to one on SERVER.  This works this way:

 ssh -NfL CLIENT:SERVER MIDDLE

CLIENT is generally “[localhost:]port”, SERVER is the destination (remote) “host:port”, and MIDDLE is the far end of the tunnel (generally “user@host”).  The “-N” option says don’t run a remote command; if you don’t specify a command ssh tries to run a login shell.  “-N” suppresses this and only creates the tunnel.  (You can forward ports even without this option.)  “-f” says to put ssh in the background, useful for a proxy server.  Note that SERVER is resolved from MIDDLE.  That’s why SERVER is often just “localhost:port”, when you don’t need the gateway host in the middle.

This creates a listening (server) socket on the CLIENT, listening for arriving IP packets sent to port port.  localhost is the default, so CLIENT is usually just a port number (but you can use “*” to indicate the socket should listen to IP packets arriving from any interface, to port port.)   If the remote host is running sshd, you can use the server’s hostname for middle and “localhost:port” for SERVER.

Consider this example:  You want to point your web browser to http://wpollock.com/, but this is blocked on your host (the CLIENT).  However it is available from YborStudent, another host where you have ssh login ability.  You need a tunnel from your localhost to YborStudent, which should then forward the packets to wpollock.com to port 80:

  ssh -NfL 9000:wpollock.com:80 wpollock@YborStudent

This will create a tunnel from localhost to the sshd running on YborStudent.  Any TCP connection to locahost:9000 will be forwarded to YborStudent, where the sshd server there (acting as a proxy client) will forward the packet to wpollock.com:80.  Any response is returned over the same tunnel.  (Test by pointing your browser to http://localhost:9000/.)

For example:  ssh -NL 110:mailserver:110 remote tells the SSH client to take any traffic addressed to localhost port 110 and shove it through a secure tunnel to the SSH server running on remote.  Upon receipt, remote’s sshd forwards the traffic to host mailserver port 110.

Qu: what does this do: ssh -NL 7777:localhost:80 wpollock.com

Ans: Forwards localhost:7777 to wpollock.com:80.  (Note the “localhost” for SERVER is a DNS name from MIDDLE’s point of view!)

When done you tear down the tunnel by using kill on the PID of the (background) ssh that was started on CLIENT.

The -R option permits reverse port forwarding from remote hosts, through the SSH server, to the SSH client.  This works by creating the listening socket (the proxy server) at the remote end, and the proxy client at your end.  You might use this to make a web server inside a secure LAN appear to be a service on somehost in the DMZ; create the reverse tunnel from the web server to somehost.

Another good use of the -R tunnel, is to allow someone to ssh into your system (think “tech support”).  You could, for example, allow your instructor to ssh into your home computer.

Difference Between Local And Remote Port Forwarding

-L is mnemonic for “local”.  If I log in to CLIENT host and then run:

    ssh -L 7777:localhost:110 SERVER

Then SSH creates a new port number 7777 on the CLIENT.  If I connect to localhost:7777 from CLIENT, then I am actually talking to SERVER:110 via a SSH tunnel.  (And the program listening at SERVER:110 thinks the connection came from localhost (i.e., 127.0.0.1 from SERVER).  But if I had run:

    ssh -R 7777:localhost:110 SERVER

Then SSH creates the new port number 7777 on the SERVER (not the client).  Now if I log into the SERVER and connect to its port 7777, I wind up talking to port 110 on the CLIENT.

So, -L creates a local port that allows me to access a remote service.  -R makes a local service available to the remote machine on a port that “looks” local to the remote system.

SSH-Based Virtual Private Networks (VPNs are discussed in more detail on p. 334)

OpenSSH contains support for Virtual Private Network (VPN) tunneling using the tun(4) network pseudo-device, allowing two networks to be joined securely.  The sshd_config(5) configuration option PermitTunnel controls whether the server supports this, and at what level (layer 2 or 3 traffic).

The following example would connect client network 10.0.50.0/24 with remote network 10.0.99.0/24 using a point-to-point connection from 10.1.1.1 to 10.1.1.2, provided that the SSH server running on the gateway to the remote network, at 192.168.1.15, allows it.

On the client:

# ssh -f -w 0:1 192.168.1.15 true
# ifconfig tun0 10.1.1.1 10.1.1.2 netmask \
     255.255.255.252
# route add 10.0.99.0/24 10.1.1.2

On the server:

# ifconfig tun1 10.1.1.2 10.1.1.1 netmask \
     255.255.255.252
# route add 10.0.50.0/24 10.1.1.1

Client access may be more finely tuned via the /root/.ssh/authorized_keys file and the PermitRootLogin server option.  The following entry would permit connections on tun device 1 from user “jane” and on tun device 2 from user “john”, if PermitRootLogin is set to “forced-commands-only”:

 tunnel="1",command="sh /etc/netstart tun1" ssh-dsa ... jane
 tunnel="2",command="sh /etc/netstart tun2" ssh-dsa ... john

Since an SSH-based setup entails a fair amount of overhead, it may be more suited to temporary setups, such as for wireless VPNs.

Another option is stunnel, such creates SSL tunnels instead of ssh tunnels.  To use you create a config file, run stunnel with it.  That creates a local proxy that can accept plain text in, and SSL out (i.e. http://localhost:123 --> https://remote/).

Here’s an example for ssh encrypted POP email access (Not yet correct):

$ ssh -S -N -L 54321:popserver.example.com:110 ssh-server.example.com
$ pine
??? ssh -gfnL 54321:localhost:110 localhost server-cmd

Then configure your MUA (or fetchmail) to get POP3 email from port 1234 from ssh-server.example.com, and voila!  POP3 passwords and all the email is now sent encrypted.  Note some port numbers require root privileges to access.

SSH to Encrypt VNC Session

With port forwarding and an sshd under Cygwin (Unix tool set ported to DOS), you can safely tunnel VNC sessions without relying on a SSL/TLS wrapper.  VNC works for Unix, Linux, and Windows.  First, create an SSH tunnel to the VNC server running on the remote machine:

    $ ssh -L 5900:localhost:5900 user@remotebox

Then, in another window start the VNC client:

    $ vncviewer -encoding tight localhost:0

The trick is remembering that :0 means 5900, :1 means 5901, etc (just add 5900 to the display number).  Many clients assume you’re specifying an implicit port number if you go much over :20.

You can use stunnel instead with VNC, and use an SSL aware VNC client.

The TightVNC client doesn’t support SSL, though newer client/server pairs that are platform specific like UltraVNC (for Windows) do.

Securing sshd

To use make sure the sshd daemon program is started at boot time.  It can take a while to generate the random keys used, so sshd should not be started via xinetd.  The configuration files are in /etc/ssh/*, especially sshd_config.  (Controls root login permitted, banner display, keep-alives, ...)

Some of my servers have been under attack for many months now.  Sometimes twice a day, hundreds to thousands of SSH connections are attempted using a dictionary of common usernames.  I have been saving the logs but no pattern has emerged yet from the IP addresses.  Of course they don’t get in (yet), but I’d like a more automatic way to respond to such attacks.  Some lessons learned:

·       Have account naming policies that forbid usernames that commonly appear in dictionary attacks.

·       Make sure all system accounts are disabled, and where possible have invalid shells (such as /bin/false or /sbin/nologin).

·       Never allow sshd (or other) root logins.  For one thing, many older versions of ssh (OpenSSH) had a vulnerability where the length of the password could be determined.  That makes it much, much easier to crack a password.  Second, restricting root login gives you an audit trail on who became root.  Third, having to crack two passwords to get into your system is much harder than one, and gives more time (and log data) to discover the attack (rings of security).

·       Make sure you protect your SSH to allow only “key” (not password) access, or at least minimize the number of valid users.

·       On production servers, only a few users should be authorized to access the server.  Use AllowUsers to restrict logins from only those users.  If some package/whatever creates an account and I don’t know about it, it still can’t be exploited.

·       Use iptables rules only to allow SSH for a handful of addresses.  Most servers only need to allow access for a very limited range of IP addresses.  Mobile users can store keys on a flash drive (along with PuTTY).  You can (reasonably) safely allow key access from anywhere.

·       Configure the PAM “su” policy only to allow a few select users to succeed with the su command.  (The members of group “wheel”.)  Other users who attempt “su” will fail even if they know the password.

·       Change sshd from port 22 to some other (unadvertised) port.  This will stop only the most casual attacks (a port scan will reveal your new ssh port), but a lot of attacks today are casual.

·       Use a PAM module that looks up the user’s cell phone number then sends an eight-digit random number to the user’s cell phone, which the user has to type in at the login prompt.  This is a variation on the theme to require human interaction (and thus stop automated login attempts).  A similar idea is to direct the user to a web page, where they must fill out some form and then re-try the ssh access.  The form could require the entry of some data seen in an OCR-resistant GIF.

sshd Configuration

The two files are /etc/ssh/ssh_config, which is used to configure global client options (the user can override these with a ~/.ssh/config file, or from the cmd line).

The server is configured from sshd_config file.  See the man page for the full list of options you can configure.  Some interesting ones include:

·       *Authentication (e.g., PasswordAuthentication), controls which authentication mechanisms can be used.  For highest security disable all except PubkeyAuthentication, which requires public key access only.

·       AllowUsers, Specifies which users can login via SSH.  There is also a deny directive, as well as ones for all/deny groups.

·       Protocol, specifies the protocol versions sshd supports.  Valid choices are “2”, “1”, “2,1”, and “1,2” (allow both, but first one listed is preferred).  Use “2” for highest security;

·       PermitRootLogin (defaults to yes!  Change to “no”);

·       StrictModes (specifies whether sshd should check file modes and ownership of the user’s files and home directory before accepting login);

·       VerifyReverseMapping, specifies whether sshd should try to verify the remote host name;

·       LoginGraceTime, the server disconnects after this time if the user has not successfully logged in.  Default is 2 minutes, “0” means disable feature.

Add “sshd: ALL” to /etc/hosts.allow.  In real life, you should limit where incoming SSH connections are allowed from!

Don’t forget to also check the firewalls (host, routers) to allow SSH to this host.

Updated /etc/ssh/sshd_config  (grep -Ev '^#|^$' sshd_config):

Protocol 2

SyslogFacility AUTHPRIV

PermitRootLogin no

PasswordAuthentication yes

ChallengeResponseAuthentication no

GSSAPIAuthentication yes

GSSAPICleanupCredentials yes

UsePAM yes

X11Forwarding yes

KeepAlive yes

ClientAliveInterval 20

ClientAliveCountMax 3

PermitUserEnvironment yes

Banner /etc/issue.net

Subsystem     sftp  /usr/libexec/openssh/sftp-server

The “PermitUserEnvironment yes” lines enables processing of ~/.ssh/environment file, to set PATH, umask, etc.

(Changes from default config: PermitRootLogin, X11Forwarding, KeepAlive, ClientAlive*, PermitUserEnvironment, and Banner.)

Advanced Client Configuration

A use can override the system defaults from /etc/ssh/ssh_config, with ~/.ssh/config.  This file permits some very useful configuration settings; see ssh_config(5) for details.  With this file, you can create aliases for hosts and also specify usernames and other settings per host, so you can type in much less on the command line.

One useful setting is “ForwardAgent”.  If you set that, and are running the ssh_agent locally, then when you ssh to the remote site, the agent is started there as well.   So you can ssh hostA, then from there, ssh HostB without re-entering the password/passphrase for the secret key.

Lecture 19 — SELinux (MAC)   [Also discuss other Linux MACs ]

SELinux (Security-Enhanced Linux) is an implementation of Mandatory Access Control (MAC) in the Linux kernel using the Linux Security Modules (LSM) framework.

Standard Linux security is a Discretionary Access Control (DAC) model.  In this standard Unix/Linux DAC model, file and resource decisions are made solely on user identity and ownership of the objects (files, devices, and ports).  Each user and program/process run by that user has complete discretion over the user’s objects.

A MAC system addresses some of the DAC problems.  With MAC security, you administratively define a security policy.  The policy describes the access permissions for all subjects (users, programs, and their processes) to all objects on the system.  Decisions are based on all the security relevant information available, and not just authenticated user identity.

The MAC policy cannot be changed without a reboot.  Even root or a kernel module can’t bypass this security!  (If loaded in a testing mode, root can turn enforcement on or off with the setenforce command.)

The Linux kernel supports a number of security related sub-systems: fine-grained rootly powers (capabilities), a traditional DAC security system, an LSM (loadable security module) that supports one (fine-grained) MAC or another, and an audit system to log (among other things) MAC messages.  (The Linux auditing system is discussed in a later section.)  Let’s see how these sub-systems interact to provide a very secure system.

The SELinux MAC system is designed to enforce the confidentiality and integrity requirements of a system security policy.  It is designed to prevent processes from reading data and programs, tampering with data and programs, bypassing application security mechanisms, executing untrustworthy programs, or interfering with other processes in violation of the system security policy.  It also helps to confine the potential damage that can be caused by malicious or flawed programs.

SELinux provides two additional security systems that are rarely used.  One is a role-based access control, where you can define various roles, which users can assume which roles (via the newrole command), and what can be done by a process run with a given role.  (sudo is more commonly used for this.)

MLS

SELinux can also be used to implement a MLS (multi-level security) system.  A MLS allows an administrator to assign security levels to files, such as unclassified, secret, most-secret, top-secret, read-this-and-die, ...  A given user is also assigned a security level.  Then access is denied if your security level is less than that of the file.  (See RHEL 7 SELinux, the MLS section.)  Note that Red Hat provides a special SELinux policy for MLS.

MCS

SELinux also supports MCS (multi-category security).  MCS allows the assignment of MCS labels to users (processes) and files.  With MCS, it isn’t sufficient for a process to have an MLS level equal or greater than the file’s level; the process must also have been assigned the same category as the file (or the file has no MCS category defined for it).  It is more commonly used than MLS; it works well for virtualization (by assigning different categories for each VM or container, the processes within that VM or container can access their files, but no others).  (See CentOS MCS, sVirt.)

A recent article by Dan Walsh (who knows SELinux very well) compares the various enforcements possible, well-illustrated, can be found at OpenSource.com.

Enabling SELinux may add about 7% performance penalty.  Note most backup tools don’t currently know about the SELinux labels on files, which are stored on most types of filesystems as extended attributes.  (Use star or other modern tools to back these up.)

A flow chart of a typical system call: check root if required capability (rootly power) was granted to the process, check MAC policy (only if an LSM is installed), check DAC.  Finally, if turned on (is by default), generate audit message.  If allowed by capability check, MAC, and DAC, then do task.

Course-grained versus Fine-grained Control

Another problem addressed by SELinux is that traditional kernel security systems are too course-grained.  As an illustration (nothing to do with SELinux), consider traditional rootly powers.  In order to perform a privileged action a process had to have a EUID of 0 (root).  In fact, the flow-chart for kernel security looked similar to:

If UID=0 allow.
else if DAC says OK, allow.
else deny.
Generate audit message.
If allowed then do task.

For example, only root can open a server (listening) socket on a privileged port (any port <1024).  So your web server must launch as root.  But this also allows your web server permission to halt the system or remount partitions.  This is too much privilege!

Note the traditional DAC security system is considered course-grained.  For a given user there are only three bits (R/W/X) that can be set!  For example, to allow a user to view their password aging information in the /etc/shadow file, with DAC you need to allow all users Read permission on the whole file.  Obviously a terrible idea!  (Currently, passwd, chage, and others are either setUID to root, or use a “helper” program that is setUID.)

With SELinux you can assign the finest possible level of control.  You can specify which set of system calls are allowed for every file (and other objects, such as sockets), for every combination of user and program!  With this level of control, you can indeed allow any user read permission to /etc/shadow only if they are using the chage program.  The modern (2015) flow-chart of a typical system call is:

If the process has the required rootly capability, allow.
If DAC says okay, allow.
If MAC installed and MAC says okay, allow.  (Note SELinux in permissive mode will allow, but generate a denied audit message anyway.)
If auditing is enabled, generate an audit message.
If allowed by capability check, DAC, and MAC, then do task.

Checking for a capability in the kernel code is as simple as this:

if (!capable(CAP_SYS_NICE)) // old: if (!suser())
   return EPERM;

(The DAC and MAC checks are equally simple.)

Reducing policy size to a manageable level

The only problem with such fine-grained control is that you need to specify so much.  A default rule of deny is used but you still must list every user (system and human login), every program on the system, and for each list all system calls allowed for each and every file on the system.  This will lead to many millions or tens of millions of rules in the policy for a general-purpose system!

SELinux allows many simplifications to reduce the size of the policy.  For example, many users and many programs have identical security implications.  So rather than listing every user you can assign the same user identity to multiple users (typically just one, user_t, is used for non-system accounts).  The same is true for programs.  The security implications of the echo or expr commands is the same, so a single security label can be used (e.g., bin_t).  Finally many or most (data) files can be treated the same, so a single label can be used for these (e.g. user_home_t).

Additionally you can define (or rather tweak the default) rules for assigning security IDs to new users, and security labels to new files.  So there is no reason to continually update the policy.

There are pre-defined policies for common scenarios you can use; it is rare to need to write one.  The NSA distributes one called strict, generally used for server-only systems, and Red Hat distributes one called targeted, useful for home systems (general use, plus a few services).

In additional to providing the tools to tweak or write your own policy, SELinux supports conditional policy rules.  These allow a policy to be customized a bit without any change to the policy itself.  For example, suppose you are running a web server.  Some servers need to support user directories, but by default the HTTP_T user has no access to /home/*/public_html.  Rather than force you to modify the policy to allow user directories, you can take advantage of the conditional rule used on the targeted policy for this.  You set a Boolean flag to true to allow this, or false to deny.

Depending on which conditional rules a policy has, the set of Booleans available to the SA can vary; the Fedora 7 targeted policy has 80 Booleans.

The set of Booleans supported by your current policy can be viewed with getsebool -a (see below for more detail).  Change them with setsebool (or other tools).

For a list of Booleans, an explanation of what each one is, whether they are on or off and their default state, run the semanage boolean -l command as root.

LIDS and AppArmor are other MACs you can use instead of SELinux.  They are much less fine-grained than SELinux but sufficient for most systems.  LIDS uses an iptables-like syntax to define rules for the policy.

User identity

With traditional *nix DAC, a user identity (and set of groups) is assigned to a process when it is created.  But by using various system calls this UID can be changed.  A malicious program can change identities to get around DAC security.  Also audit logging is fairly useless if it only shows “root” did this and “root” did that.

With SELinux, a process is assigned an identity when created that cannot be changed for the life of that process.  This means changing a UID won’t defeat the MAC security, and the audit log will always show the true user.  (Use id -Z to view.)

The one exception to this rule (of not changing the user identity) is the login program.  When run, it initially is run by the root user.  After a successful login, it must exec a shell for the user.  At this time, the user identity of the new shell is not root, but rather the identity of the user who just logged in.

Implementation

Checking the policy for every access to a system call would be very slow.  Instead SELinux caches the access decisions.  When a system call is invoked, a check of this cache is done.  If the list (called a vector because it is cooler-sounding) of allowed access for this user/program is not in the cache, the system will pause, calculate it, add it to the cache, and retry the lookup.  It is then a simple matter to search this vector for allowed access to a particular file/socket/whatever.

Because of this architecture, the SELinux “permission denied” messages show up in the log as AVC (access vector cache) messages, for example (one long line):

type=AVC msg=audit(1184605652.582:39): avc:  denied  { search } for  pid=2570 comm="procmail" name="root" dev=sda10 ino=2301537 scontext=system_u:system_r:procmail_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=dir

The results of the mount command show sda10 is the / (root) filesystem, and the results of:    # find -H / -xdev -inum 2301537

shows that procmail was denied search access for the directory /root.  Most likely, it was looking for /root/.procmailrc or /root/.forward.  A properly secured system won’t have those files, so it is no problem blocking procmail from looking for them.  If you want to allow procmail this access, first add a+x (search) permission to /root.  This may take care of the problem.  If necessary check if there is a Boolean you can set.  If not you must tweak the policy by adding the new allow rule(s).

Policies

MAC under SELinux allows you to provide granular (very precise) permissions for all subjects (users, programs, processes) and objects (files, devices, ports). You can safely grant an application just the permissions it needs to do its function, and no more: you specify for each combination of user and process, which system calls are allowed to each file, device, socket, and port.  However this can be very complex to setup correctly.  Normally you use a pre-defined security policy that will suit your needs.

Initially a strict policy from the NSA was enforced.  However whenever new users, devices, or software is added to the system, or to conform to local policies, a local security expert is needed.  For most home and school uses of Linux this was felt to be too restrictive.

Fedora now ships with both this strict policy and a targeted policy (each in a separate RPM package).  The targeted policy labels most standard daemons with an appropriate domain (such as named_t for named) and the rest of the system processes run in “unconfined_t” domain, which allows anything.

Note that even if the MAC allows something, standard DAC security still applies.  Thus some operation must be allowed in both before it can be done.

The security policy includes such things as

·       The TE Matrix (the table which controls access)

·       A list of roles various users (login accounts) can use

·       A list of types for specific files (those in /bin, /sbin, /etc, /dev, ...)

·       Other policy rules, such as which roles a user is allowed to use

·       Defaults, such as the default role for new user accounts you add, and the default type for new files created (e.g., files created in a user’s home directory get labeled one way, files created in /dev  or /etc might get different defaults).

·       A set of on/off flags called Booleans that affect the policy enforcement.

The targeted policy defaults are in /etc/selinux/targeted/contexts in the file files/file_contexts.

You can define which policy you will run by setting the SELINUXTYPE environment variable within /etc/selinux/config.

Concepts of roles, domains, types, and process contexts

The SELinux implementation uses role-based access control (RBAC), which provides abstracted user-level control based on roles and Type Enforcement (TE).

The domain of a process determines what access that process is allowed.

Every user is assigned a role.  A role determines what domains can be used for a process.  The domains that a role allows are predefined in the policy.  If the current role does not authorized the process to enter some domain (in the policy database), then the transition to the new domain will be denied.  The policy lists what roles a given user is allowed to use.

Every process is assigned a security context that includes the user identity, the role of the user, and a domain.  To keep the tables as simple as possible SELinux uses generic user IDs in the rules.  So you will not usually see your UID but rather “user_t” in the various tools that show security information.  You may also see “system_u” (for kernel processes) and “root” (used in the strict policy).

Recall that on Unix/Linux systems, every process is started by another, the parent process.  When starting a new process the system must determine its security context.  This is based on the parent’s security context and the type assigned to the program file.  The initial security context is the same as the parent.  SELinux then tries to change the domain of the new process according to the current security context and the type of the file.  This is called a domain transition.

For example, if you try to run the passwd program from your shell prompt, the shell will start a new process from the /bin/passwd file.  The security context of the new process will have the same identity and role of the parent (bash) process, but the domain will be “user_passwd_t”. 

In order to allow a user from the unconfined_t domain (the unprivileged user domain, called “user_t” in the strict policy) to execute the passwd command, the following is specified in the relevant config file:  “role user_r types user_passwd_t”.  It shows that a process with the user role (user_r) is allowed to enter the user_passwd_t domain, i.e. it can run the passwd command.

Each file and other object (devices, sockets, ...) is assigned a type.  The security context of a file can vary depending on the domain that creates it.  By default a new file or directory inherits the same type as its parent directory.  However the policies can be set up to do otherwise.

A table called the TE Matrix determines what access is permitted for each combination of domain and type.  This is how SELinux enforces access control.

  A cross-reference on the matrix defines their interaction.  (I.e., who is allowed to do what operations on which files/devices/ports.):

For example, the domain “user_passwd_t” permits a user write access to the /etc/shadow file, but the “unconfined_t” domain (the domain of the shell) does not.

In the real world, the security needs of different organizations will be similar but slightly different.  For example, a web server may need to run CGI scripts but another may not need that access.  Rather than make many policies, only slightly different from each other, each policy supports a set of conditional rules in the TE Matrix.  These permit or deny access according to the current value of a Boolean.  So a single policy may permit or deny CGI access to httpd, based on some setting.

A High-Level Overview

The security context for processes started at boot time is determined by the file /etc/selinux/targeted/src/policy/initial_sid_contexts.

A user logs on and the user identity is determined.  If there isn’t a specific identity listed in the policy matching this user ID, then the identity defaults to “user_u”.  The user identities are listed in the files /etc/selinux/targeted/src/policy/users and .../local.users.

Next, the user is assigned a role.  If one isn’t listed for that user in the policy files, the default is used (“user_r”).

The login shell then gets assigned a security context that includes a domain, the user id, and the role. 

When the user runs a command, the new process is assigned a domain based on the context of the parent process (user, role, and domain), as well as the type assigned to the executable file.  If the process is assigned a different domain than the parent process, this is called a transition.  (The role determines what transitions are allowed.) The context of your login shell is a transition from login process’s role, domain of system_r:remote_login_t to system_r:unconfined_t.

As the process runs, it will execute system calls.  (For example, suppose the process attempts to read from a file.)  After normal permissions checking, the kernel checks to see if a LSM such as SELinux is installed.

If SELinux is installed (enabled), it gets called to check if the access is permitted.  The check is based on the domain of the process, the type of the file being accessed, and the specific system call being made.  If the type of the file is unknown for some reason, a default type is used.

If SELinux is enabled a log record is made of the check.

If the access is not permitted by SELinux and SELinux is in enforcing mode then the access is denied.  If SELinux is in permissive mode then the access is allowed by SELinux even though the rules don’t allow it.  (Note you still get the log record).

Labeling

The process’s domain lists what that process can do with various objects (or types of objects).  An example is an ordinary user running the date command.  Such a domain permits the process to use gettimeofday(2) but not settimeofday(2) system calls, and to read(2) certain files such as /etc/localtime (the system time zone setting), but not others.  The domain is different if root runs the date command.

There are three types of domains: unconfined allows the process to run as if SELinux were turned off; permissive is the same but logs violations anyway; and confined which is the default, enforcing domain.  Note when you run a program, or init or xinetd does, a transition to a different domain (process label) may occur.

Demo: ps -Z & date  (then repeat as root)

To make this scheme work every object and user on the system must be labeled with its type.  The use of roles and defaults (e.g., files in a directory can inherit the context of the directory) make this simpler.  You can’t do this with setfattr, as unrestricted use of that would allow any user to bypass the security policy just by re-labeling things.

(Demo: touch ~/foo /tmp/foo; ls -Z ~/foo /tmp/foo)

Setting the security context on files is done with the setfiles command.  This command is complex and Red Hat has added a simpler command, fixfiles.

Try as non-root: fixfiles check . |less (this checks the labels against the policy defaults).

You can use the option “restore” to fix any files that aren’t labeled or are labeled incorrectly.  Using the “relabel” option instead ignores any existing labels and relabels the files according to the policy.  As root you can simply run “fixfiles restore” to fix the labels on all mounted filesystems, which can be handy with removable media:

fixfiles relabel
reboot

Another approach is to take advantage of the /.autorelabel mechanism:

touch /.autorelabel
reboot

(In addition to setfiles and fixfiles you may find restorecon.  All these commands do roughly the same thing.)

If the defaults aren’t correct for some file, you can change the security context.  To change the security context of any file use chcon command.  Such a change will be lost if the file is relabeled from the policy.  A permanent change should be made in the policy itself.

Users can’t be relabeled with a different identity.  However you can assign different roles to a user at login time, or using newrole command.  This command starts a new shell with the process context modified by the command line arguments (if the policy permits).

The policy naturally prevents regular users from increasing their privileges this way and not even root can change identity, but you can change the role for the process or its domain (type).  Note that change the role will only affect further processes started by that one; in the case of a command such as date that doesn’t start additional processes there should be no effect.

Implementation

Initially, the Linux kernel needed to be patched for SELinux, and re-built.  Instead, a change was made to how the kernel implements system calls:

userland program--> kernel API--> check DAC, do auditing, if LSM loaded, invoke LSM MAC check, if still allowed, do the task and return else generate audit msg.

SELinux is implemented as a LSM.  LSM (Linux Security Modules) are an extension of the Linux kernel, which allows security systems to be easily added to the kernel.  SELinux is just one possible security system that can be added.  Another is LIDS.  (Only one LSM can be loaded in the kernel at a time.  Initially no LSM is loaded, and thus the MAC check is a no-op (does nothing).

The context or type of a file is stored in the filesystems using extended attributes.  Each file contains a set of (zero or more) attribute-value pairs.  Attributes have hierarchical names such as security.selinux, which is the attribute where the security context of a file is stored.  (SELinux requires not only LSM in the kernel, but patches that support extended attributes.)

(Note that the original version of SELinux didn’t use extended attributes, but stored this information elsewhere.  To map users and objects to this table, special security identifiers (SIDs, PSIDs) were added to the system.  These are no longer used.)

Normal attributes vary from one filesystem to another, and are examined and changed with lsattr and chattr commands.  Extended attributes are supported much the same way as ACLs are.  Most filesystems today support extended attributes (and ACLs) but not all.  Use the commands getfattr and setfattr to examine and changes the extended attributes of files:

          getfattr -m . -d file

(Normally this command only shows extended attributes whose name starts with “user.”.  To see other attributes use the “-m RE” option to specify a regular expression to match what attributes to show.  A dot will match all. The “-d” means to dump the attributes.)

Many traditional commands that show file information have been retro-fitted with new options, to show the security context of files.  Usually the option is “--context” or “--lcontext” or “-Z”.  (Show: ls -Z file and id -Z).

The pseudo-filesystem /sys/fs/selinux provides current information from the kernel about SELinux.  This filesystem is similar to /proc.  The /etc/fstab file should include the following:

none /sys/fs/selinux selinuxfs noauto 0 0

(Under older systems, /sys/fs/selinux has been moved to /selinux.)

The actual configuration files are collected under /etc/selinux.  There is a config file there, as well as a subdirectory for each installed policy.  In that directory you will find further configuration: the Booleans, which are on/off flags for various aspects of the policy such as which daemons are allowed unrestricted access (i.e., for which daemons have you turned off SELinux enforcement), the actual policy rules (the TE Matrix), and the default assignment of labels/contexts to objects (roles, users, ...).

Summary

All users get assigned a role based on the policy; if no roles are listed for some user a default is used.  A user can assume a new role by starting a new shell via the newrole command, or by changing the policy (and logging in again).

All files get a type based on the policy; if no type is listed for some file (or directory), a default is used.  The type of new files depends on the type of the parent directory and the process’s domain.  A user can relabel files, either by changing them directly or by edited the policy and then relabeling.

Every process gets a domain based on the security context (user, role, domain) of the parent process, and the type of the program file.  If the process’s domain is different than that of the parent process, a domain transition occurs.  The definition of roles in the policy determine which transitions are allowed.

Access is controlled by the type enforcement matrix, which is a table with one row for each domain and one column for each type.  The cell in the intersection of a domain and type lists what system calls are allowed.  The matrix is stored in binary files, which are compiled from plain text files you can edit.

SELinux is implemented using Linux Security Modules interface of the kernel, and stores type information using extended attributes in the filesystems.  All the components of the policy (the definitions of the roles, the assignment of roles to users, and the TE Matrix) are stored in a number of files under /etc/selinux/policyname.

Linux currently ships with two policies, strict and targeted.  The targeted policy is the default, and is suitable for desktop users (essentially only certain daemons have access enforcement) and learning.  In this policy you can change the mode, update and reload the policy, and even turn SELinux on and off, without rebooting.

Policies can be edited (or created from scratch) using the policy source files, which then get compiled into the (binary) policy that can be loaded.

Rather than edit the policy source directly, each policy supports a set of Booleans, which selectively turn on or off enforcement some rule(s) in the policy.

Using SELinux — Turning it on and off

Displaying the current status can be done with the commands sestatus [-v] or getenforce, or “cat /sys/fs/selinux/enforce”.  In permissive mode SELinux will check and log all accesses, but won’t prevent any.  To switch the modes without rebooting:

    echo "1" >/sys/fs/selinux/enforce

which will switch the kernel into enforcement mode.  Conversely,

    echo "0" >/sys/fs/selinux/enforce

switches into permissive mode.  (The Red Hat command setenforce does the same thing.)

Add selinux=0 to your kernel boot line to disable SELinux at boot time, “1” to enable.  Use enforcing=0 in your kernel boot line to boot into permissive mode, “1” for enforcing mode.

The mode can also be set by the /etc/selinux/config file, which overrides the boot parameters.  The config file determines if SELinux is disabled completely, enabled in enforcing mode, or enabled in permissive mode.

This file can also set the environment variable SELINUXTYPE to determine which policy to use.

Any files you create while SELinux is disabled will not have SELinux context information.  You may need to relabel whole file systems and it’s even possible you will be unable to later boot with SELinux=1, requiring a boot to single-user mode for recovery.

Using SELinux — Modifying the Policy

Use getsebool -a to see what Booleans the current policy supports, and setsebool to change them (or “ls /sys/fs/selinux/booleans” to list and echo 0 or 1 to /sys/fs/selinux/booleans/name to change.)  You can examine the current policy using sesearch and seinfo.  (May require additional packages to be installed such as setools-console and selinux-policy-doc.)

The roles a user can assume is determined by the policy, in the files /etc/selinux/policyname/users/*.  You can add additional roles for a user (or change the default role for a user) by editing the local.users file.

To modify the policy rules that make up the TE Matrix requires that you installed the source package for the policy.  You then edit the text files to make changes, or use certain tools to generate the text to append automatically (audit2policy).  Next you must re-compile the policy source using the checkpolicy tool:

cd /etc/selinux/targeted/src/policy
edit the policy files
make
make relabel
reboot

Another tool called load_policy can be used to replace the old policy with the new one.  Naturally this is dangerous!  Red Hat added this tool because typically users are testing and debugging policies, but on a production system, not even root should be allowed to run this command; the policy should only be loaded at boot time.  You can prevent users (even root) from turning SELinux on or off, or changing the mode, by editing the file /etc/selinux/policy/src/policy/macros/admin_macros.te and remove (or comment-out) the following lines:

    can_setenforce($1)
    can_setbool($1)
    can_setsecparam($1)

If you comment out these lines, recompile, and reload the policy, then it will not be possible to change the enforcing mode (not without changing the policy at least, and you could probably block that as well).

The policy file can be modified by root by default (shouldn’t allow that on a production server) but this is beyond human ability.  Newer SELinux tools allow you to create local policy modules that over-ride the regular policy.  Use audit2allow and audit2why to figure out the rules you need, or learn to use the other se* tools.  Report the problem to Red Hat (or your vendor) so they can update the policy or add a Boolean.

Use semanage to make changes such as managing the labels assigned to users, ports, devices, or other hosts, managing booleans, user roles, etc.  Use semodule to create local policy modules that can over-ride the compiled policy.

The semanage command is pretty slow.  It can take 10-20 seconds for a semanage command to complete.  semanage recompiles a huge amount of policy.  In Fedora 15, there are nearly 500,000 allow and dontaudit rules.  The compiler checks each type, user, role, etc. to make sure they are valid.  Executing multiple semanage commands in RPM post-installscripts can be very slow; these can be batched, but not all RPM authors bother.

Using SELinux — Backups

Use the (recent version of) star command to backup files with their SELinux contexts.  For example,

          star -xattr -H=exustar -c -f output.tar [files]

Also the dump and restore utilities for ext[234] have been updated to work with XATTRs (and therefore SELinux contexts).

Another backup issue is restoring SELinux changes.  (This is used to make identical changes to multiple hosts, too.)  First, copy all the .pp files you’ve installed into some archive file.  Next, run the command “semanage -o mySELinux.conf” to save all the other changes in a way that can be re-applied using “semanage -i”.  Add that file to the archive too.  To restore, use semange and semodule.

Using SELinux — Trouble-Shooting Policies and Fixing Problems

Modifying SELinux Policy

What if the current policy doesn’t allow something it should?  There will be an error message in the audit log (or some log file such as messages).  Run the audit2why command to see why access was denied.  Another tool is used to automatically convert audit failure messages into an SELinux rules file.  Suppose for example, SELinux denies some action of Postfix MTA and you think it should be allowed.  Here’s all you need to do:

Step 1: Create policy:

  audit2allow -M mypostfix < /var/log/audit/audit.log

This will create a text version of the policy file, name.te, as well as a compiled version, name.pp, in the current directory.

Step 2: Enable the new policy:

  semodule -i mypostfix.pp

That’s it!

When you install name.pp, it will replace any previously install policy package file of the same name.  So if a new error occurs either (1) create a new file with a different name and install both, (2) if both errors are still in the log, just create a new policy package file and it will contain both rules, or (3) append the output of audit2allow to the existing .te file, and then compile that.

To append the new rule to some existing name.te file, run:

 audit2allow </var/log/audit/audit.log >>mypostfix.te

and compile the .te files in the current directory with:

 make -f /usr/share/SELinux/devel/Makefile

and install as before (using semodule).

The output instead could be added to the local.te file (in the directory /etc/selinux/targeted/src/policy/domains/misc/), which is one of the source files for the TE Matrix.  (If fixing a broken policy, you may wish to add to .../domains/programs/xyz.te instead.)  After that, “make policy install load” to rebuild the binary TE Matrix (security policy) file, and reboot.  This requires you to have the source of the policy installed (a separate RPM).

Using SELinux Booleans

The SELinux policy can include conditional rules that are enabled or disabled based on the current values of a set of policy Booleans.  These policy Booleans allow runtime modification of the security policy without having to load a new policy.  These are true (1) or false (0) settings. 

For example, the Boolean httpd_enable_cgi allows the httpd daemon to run CGI scripts if it is enabled.  If the administrator does not want to allow execution of CGI scripts, she can simply disable this Boolean value.

Use the setsebool command to change them.  (The “-P” option will update the Boolean file on disk as well.)  Use getsebool to view the current values.  (Demo “getsebool -a |less”.)  See also man page for booleans(8).

Using SELinux — Debugging Daemons

SELinux intentionally disables access to the tty devices, to stop daemons from communicating back with the controlling terminal.  This communication is a potential security hole because such daemons could insert commands into the controlling terminal.  A broken or compromised program could cause serious problems with this.  Normally daemon programs don’t produce any output, but you may need to run them with debugging mode on.

There are a few ways you can capture STDOUT from daemons.  One method is to pipe the output through the cat command:

snmpd -v | cat

Many daemons have special man pages describing SELinux issues and how to overcome them.  See for example ftpd_SELinux(8), named_SELinux(8), httpd_SELinux(8), nfs_SELinux(8), and samba_SELinux(8).

Case Study: FC4/SELinux and Apache:

SELinux was preventing access to CGI and user home directories (UserDir).  To fix, I first tried the instructions in the httpd_SELinux man page:

  setsebool -P httpd_enable_cgi 1

  restorecon -R ~wpollock/public_html/

  chcon -R -t httpd_sys_content_t ~wpollock/public_html

but I realized I’d have to do this for every user.  And it didn’t work!  I must’ve done something wrong or omitted some step.  Re-reading the man page reveals I also should have:

  setsebool -P httpd_enable_homedirs 1

Note CGI script files need a different label.  To set this up correctly you should change the label on all the files, and also on the directories so new files created in there will get the correct labels by default.

Although I could have used audit2allow to try to repair the policy, instead I ran the following:

  setsebool -P httpd_disable_trans 1

which prevents the httpd program from transitioning to the httpd_t domain when it starts up.  This effectively disables SELinux checks for httpd.

To examine the current Boolean values for httpd, you can take advantage of the fact that they are all named after the daemon:

   getsebool -a |grep httpd

Modern SELinux

Until 2005, SELinux tools for administrators (and policy writers) were quite primitive.  A new design has made the system much easier to use.  See fedoraproject.org/wiki/SELinux for lots of useful info on this.

Here’s a list of the most important SELinux commands you should know:

getenforce
setenforce 0 or 1 (to set permissive or enforcing)

getsebool -a
setsebool some_bool 1 or 0 (on or off)

To adjust file labels to the policy:

restorecon -vFR /some/dir

To change labeling policy:

semanage fcontext -a -t some_type_t '/some/dir(/.*)?'

For example, to add httpd file-context for everything under /web:

semanage fcontext -a -t httpd_sys_content_t "/web(/.*)?"
restorecon -R -v /web

When all else fails:

audit2why

audit2allow -alrM foo; semodule -i foo.pp

Smack [from lwn.net/Articles/244531/]

The Simplified Mandatory Access Control Kernel (Smack) is a security module designed to harden Linux systems through the addition of mandatory access control policies.  Like SELinux SMACK works by attaching labels to processes, files, and other system objects. It allows the SA to implement rules describing what kinds of access are allowed by specific combinations of labels.  Unlike SELinux, SMACK was designed specifically for simplicity of administration.  It leaves out the role-based access controls, the type enforcement, and the MLS.  So it can’t make a system as secure (in theory) as SELinux but it avoids the nightmare of SELinux administration, which is too complex for mere mortal SAs to deal with.

Smack allows an administrator to define labels for “objects” (files, processes, sockets, devices).  Labels are 1 to 7 characters in length.  (Some labels are reserved and have special meaning.)

To see if access is allowed by some task (a process) to some other object, the labels on two are compared.  By default, access is only allowed if the labels match.

There are a set of Smack-reserved labels that follow a different set of rules which allows most system objects and processes to be unaffected by Smack restrictions, so Smack doesn’t get in the way of the OS.  The SA only needs to add rules and labels for the users and processes they want to secure.

Smack uses filesystem extended attributes to store labels on files.  SAs can set labels using the attr command.  The security.SMACK64 attribute is used to store the Smack label on each file.  For example setting /dev/null to have the Smack-reserved “*” label would look like this:

attr -S -s SMACK64 -V '*' /dev/null

Users can be labeled using a patched version of ssh.  Sockets and their packets are labeled using NetLabel.  It provides packet labeling capabilities for the Linux 2.6 and newer kernel, and supports Common IP Security Option (CIPSO) labeling.

An administrator can add rules, but there is no support for wildcards or regular expressions; each rule must specify a subject label, object label and the access allowed explicitly.  The access types are much like the traditional UNIX rwx bits, with the addition of an a bit for append.

For configuration, Smack uses the SELinux technique of defining a filesystem that can be mounted, smackfs.  Typically, it will be mounted as /smack, providing various files that can be read or written, to govern Smack operation.

Smack access rules are written to /smack/load.  To change rules just write a new set of access permissions for the subject-object pair.

Here’s an example that uses the standard security levels for government documents.  Smack labels are defined for each level: UC for unclassified, C for classified, S for secret, and TS for top secret.  Then the traditional hierarchy of access is defined with a handful of rules (plus the Smack defaults):

 C  UC  rx
 S  C   rx
 S  UC  rx
 TS S   rx
 TS C   rx
 TS UC  rx

Because of the Smack defaults, UC will only be able to access data with that same label.  Because of the rules above, TS can access S, C and Unclassified data.  Also because no write permissions have been given, tasks at each level can only write data with their own label.  So secret tasks write secret data and so on.

Note that there is no transitivity in Smack rules, just because S can access C and TS can access S, that does not mean that TS can access C.  That rule must be explicitly given.

Files inherit the label of the task that creates them, with Smack ensuring that the filesystem attribute is set.  They will retain that label unless it is explicitly reset by an administrator using the attr command.

Linux Intrusion Detection System (LIDS) [www.securityfocus.com/infocus/1496]

LIDS (www.lids.org) is a Linux kernel patch that will allow users to give programs exactly the access they need, and no more.  It works much the same as SELinux, only is designed to be considerably simpler to use.

A LIDS kernel has several features.  One is a port scan detector, which is used to alert users to a possible intruder.  It logs the offending IP addresses via syslog.

The most important aspect of LIDS is to enforce strong access restrictions (ACLs).  LIDS has two kinds of ACLs: those that restrict actions that can be performed on files (such as read/write/append), and those that restrict capabilities a process may possess (e.g., changing NIC addresses, changing user ids).  Here’s an example that hides /etc/shadow form all access, except read access by sshd:

# lidsadm -A -o /etc/shadow -j DENY
# lidsadm -A -s /usr/sbin/sshd -o /etc/shadow -j READ

LIDS file ACLs are enforced as soon as the kernel boots, whereas the capability ACLs are not turned on until the kernel is sealed.  The integrity of a LIDS kernel would be suspect if inappropriate LKMs were allowed to load.  Thus LIDS allows LKMs to be loaded during startup until it is sealed with the “lidsadm -I” command.  At the time the kernel is sealed, no more LKMs can be inserted or removed from the running kernel.  (BSD has a similar capability.)

LIDS keeps its files in /etc/lids.  This is always protected by LIDS so that it is invisible (DENY) to all users including root.  Once you setup a password, you can start a LIDS Free Session (LFS).  In an LFS session, (shell) LIDS restrictions don’t apply.  You establish a LFS by using the lidsadm command as follows:

root# lidsadm -S -- -LIDS
enter password:

You now enter your password and, if correct, you are able to configure the LIDS system and override any restrictions it imposes.  Without a LFS, to make changes (such as initially setting the password) you need to reboot with LIDS off.  Add “security=0” to the boot command line.

AppArmor

AppArmor was developed by Novell allows the system administrator to associate with each program a security profile which restricts the capabilities of that program. It thus provides mandatory access control (MAC).

In addition to manually specifying profiles, AppArmor includes a learning mode, in which violations of the profile are logged, but not prevented. This log can then be turned into a profile, based on the program's typical behavior.

AppArmor is implemented using the Linux Security Modules kernel interface, the same as SELinux or LIDS.

AppArmor was created in part as an alternative to SELinux, which critics claim is difficult for administrators to set up and maintain.  Unlike SELinux which is based on applying labels to files, AppArmor works with file paths.  This is less complex to use than SELinux.  SELinux requires a filesystem that supports extended attributes, and thus cannot provide access control (currently) for files mounted via NFS.  AppArmor is file-system agnostic.

One important difference is that SELinux identifies file system objects by inode number instead of path.  This means that, for example, a file that is inaccessible may become accessible under AppArmor when a hard link is created to it, while SELinux would deny access through the newly created hard link.  On the other hand, data that is inaccessible may become accessible under SELinux when applications update the file by replacing it with a new version (a frequently used technique), while AppArmor would continue to deny access to the data.  (In both cases, a default policy of “no access” avoids the problem.)

GRsecurity

From www.GRsecurity.net, this is a set of Linux kernel patches designed to enhance security in several areas.  Some of these include: RBAC for Linux, chroot hardening, /tmp (and other publicly writable directories) hardened to prevent a race condition, auditing, non-executable stack and heap memory, kernel lockdown, prevent null pointer dereferencing, randomizing locations of data (prevents many types of attacks), /proc and related restrictions so users can only see their own processes, and many others.

These were designed for the 2.4 Linux kernel but have been ported to the 2.6 kernel.

 

Lecture 20 — Service Containment and Vulnerable/Untrusted Code Containment: chroot, FreeBSD jails, Solaris zones, Virtualization (V12N), & Clouds

Capsicum 

The Capsicum kernel support (if present) is an extension of fine-grained privileges (capabilities).  The idea is not only to limit the privileges of potentially unsafe code, but to run such code in a restricted sandbox as well.  A program such as login no longer needs access to /etc/shadow; only the small, password checking code does.  In this case, the login program is split into two parts which run as separate processes.  Only the password checking code needs the extra privilege to read /etc/shadow, and that code needs no other access at all.  The two processes communicative through sockets or another IPC means.  Capsicum works by having the privileges appear as file descriptors to the sandboxed process, passed in from the parent.

Capsicum helps but is not required to use the general technique; look in /usr/libexec for a bunch of SetUID executables that are just the privileged parts of other utilities, such as passwd.  (Modern systems may not require SetUID, but use setcap(8) instead.)

This type of privilege bracketing and containment requires extra development effort and the overhead of multiple processes (and their IPC).  Generally, the other, simpler containment mechanisms discussed next are used.

BSD Security Level

BSD also defines a securitylevel command, which can be used to freeze the system: no loadable modules will load/unload, no disk partitions can be mounted.  This level is enforced in the kernel, not even root can override once set!

Linux 2.6 includes this feature, as a loadable kernel module seclvl.  See seclvl.txt file in the kernel documentation for details.  (LIDS supports it too.)

chroot

Ordinarily, filenames are looked up starting at the root of the directory structure, i.e., “/”.  “chroot” changes the root to some other specified directory (which must exist), for all filesystem system calls (a.k.a. syscalls or APIs).  The commands run this way cannot access any filename outside of this directory.  Symlinks won’t work but hard links will, so it is best if the chroot jail is on its own partition.

This works best for a statically linked server/daemon.  You put the server and only its data and configuration files someplace, and run it as chroot.  Even if some hacker breaks into this program, they cannot access any files outside of this directory.  Not even root can break-out (in theory; several exploits are known and still possible with some systems).  (Danger: what if chroot directory contains /dev/mem or kmem or hda?  Some mount options can help.)

Not all server programs support chroot (check the man page and see if there is any command line option to enable that).  Some that do support chroot run the chroot sys call as soon as they start, others wait until completely setup before running the chroot call.  The first type requires a complete environment setup within the chroot jail; the second type can refer to DLLs and config files from their normal locations.

To set up the first type of server (say “foo”) if it uses DLLs, run “ldd foo” to see what shared objects it needs.  Then in addition to copying the application, also copy the listed files to the required positions (.../lib) under your intended new root directory.  If the executable requires any other files (e.g., data, state, device files), copy them into place too.  To find those, use:
    strace -e trace=file foo

So typically a server will have this directory structure:  .../top/{bin,lib,etc,dev}.

To write to syslog, a socket (.../dev/log) must be created within the directory structure that the program can access.  syslogd must be started with the -a socket option.)  (Demo CWS: /var/named/chroot, /var/ftp.)

Note that a trusted but vulnerable program can be started as root, set itself up, cd to /var/.../top, then execute chroot.  Thus the executable itself and initial configuration files can go in their normal locations, /sbin and /etc.  Some daemons work this way but not all; several FTP servers that (if they support chroot at all) run chroot as soon as they start, and thus need more effort to setup the chroot jail.

There is a chroot(1) command too, so you can create a chroot jail even for programs that don’t support it.  Project: setup chroot jail in ~/jail that allows dash, vim, and ls.  Install the glibc-static package (with yum), then compilea static version of hello.c:

$ gcc -static -o hello-static hello.c

Put the static binary (varify with ldd) into ~/jail.  Then run:

~ $ chroot ~/jail /hello-static

(You may need to be root, depending on your OS.)

Now try it with a dynamic binary.  You get a very unhelpful failure message.  Use commands like the fillowing to locate and copy the required DLLs:

~/jail $ mkdir bin lib etc usr

~/jail $ cp /usr/bin/cal bin/

~/jail $ ldd bin/cal
 linux-gate.so.1 =>  (0xb7773000)
 libtinfo.so.5 => /lib/libtinfo.so.5 (0x41072000)
 libncursesw.so.5 => /lib/libncursesw.so.5 (0x4f031000)
 libc.so.6 => /lib/libc.so.6 (0x4ed1d000)
 libdl.so.2 => /lib/libdl.so.2 (0x4eed0000)
 /lib/ld-linux.so.2 (0x4ecf8000)

~/jail $ ldd bin/cal | tr -s ' ' |cut -d' ' -f 1 \
   |xargs locate |xargs -I'{}' cp '{}' lib/

To find all files, use strace instead of ldd:

~/jail $ strace -e 'trace=file' env -i /usr/bin/cal \
 2>&1 >/dev/null |grep -v ENOENT |cut -d\" -f2 |sort -u
+++ exited with 0 +++
/etc/ld.so.cache
/etc/localtime
/etc/terminfo
/lib/libc.so.6
/lib/libdl.so.2
/lib/libncursesw.so.5
/lib/libtinfo.so.5
/usr/bin/cal
/usr/lib/locale/locale-archive
/usr/share/locale/locale.alias
/usr/share/terminfo
/usr/share/terminfo/x/xterm

The env -i command ignores the environment, minimizing what cal will try to open.  The grep discards files not found (and thus, weren’t needed).  You can use a xargs command similar to the one shown with ldd (but more complex) to automate this, or just copy each file manually.  Note, not all the files found are necessarily required.

FreeBSD jails

FreeBSD jails take this idea further.  The jail system call reuses chroot, and also separates out all processes (not just the one that started with chroot) to be restricted to the jail, adds networking system calls to the jail (so processes can only access pre-selected addresses and ports), removes super-user privilege to any access outside the jail (say by using a hard link, or some copy of a device file).  Combined with resource control (rctl, similar to ulimit), jails provide a high degree of isolation.

Solaris zones

Solaris 10 zones build on the jail concept and add even more security and control.  Zones are useful to run a number of applications on a single server.  The cost and complexity of managing numerous machines make it advantageous to consolidate several applications on a single larger, more scalable server, at the same time providing a lot of security.

A zone provides an application execution environment in which processes are isolated from the rest of the system.  This isolation prevents processes that are running in one zone from monitoring or affecting processes that are running in other zones, even if root.

Each zone that requires network connectivity has one or more dedicated IP addresses.  Each zone has its own set of users, names service (resolver) setup, hostname, ...  Processes that are assigned to different zones are only able to communicate through network APIs.  Zones provide virtual instances of Solaris, one per application if desired.

For efficiency, zones are managed by a single OS (kernel) instance.  Zones allow application components to be isolated from one another even though the zones share a single instance of the Solaris OS.

Solaris also supports branded zones (BrandZ) which is a zone with a separate OS.  This other OS need not be Solaris!  For example, installing the “lx” Linux branded zone would allow you to run (RHEL) Linux binaries unchanged on a Solaris server.  See virtualization below for more information.

Every Solaris system contains a global zone.  The global zone is both the default zone for the system and also the zone used for system-wide administrative control.  All other zones are called local zones.  For example, managing zones, auditing with the Basic Security Module or BSM is done from the global zone.  So is extended process accounting, and Solaris’ filesystem integrity monitor (Basic Auditing and Reporting Tool, or BART).  You deploy services from local zones, one per service.

The following shows one way to configure a local zone with access to the CD-ROM.  The method is to loopback mount the /cdrom directory from the global zone to the non-global zone:

# zonecfg -z myzone
add fs
set dir=/cdrom
set special=/cdrom
set type=lofs
set options=[nodevices]
end
#

Solaris Containers

A Solaris Container is a complete runtime environment for applications.  Using Solaris Resource Manager (to impose resource limits) on a zone forms a container.  (However, it is common to use the terms zone and container interchangeably.)  Containers provide lightweight virtualization that let you isolate processes and resources without the need of full virtualization and all the associated complexities and performance loss.

Unlike jails (which can be considered a type of zone/container), processes in containers cannot hog the CPU or RAM of the system.  They only use what has been allocated to the container.  (Resource management is optional and need not be used on some zones.)  Resource management features permit you to allocate the quantity of resources that a workload receives.

Each non-global zone has a security boundary around it.  The security boundary is maintained by:

·       adopting Solaris 10 Process Rights Management (privileges(5)),

·       name spaces (for example, /proc, /dev) isolation, and

·       allowing zones to only communicate between themselves using networking APIs such as sockets

Most software doesn’t require root privileges and can run unchanged in a zone.  Some software that does require root privileges will run unaltered, but some software needs to be examined to see if it will run in a zone.  Automated searching through the source code will catch issues, but testing with tools such as privilege debugging, apptrace(1), truss(1), and dtrace(1M) may be needed.

Process rights management enables processes to be restricted at the command, user, role, or system level.  The Solaris OS implements process rights management through privileges.  Privileges decrease the security risk that is associated with one user or one process having full super-user capabilities on a system.  The command ppriv -lv prints a description of every privilege to standard out.  (See page 189 for details on PRM.)  Rights can be granted to containers as well.

When the system creates a non-global zone, an init process is created for it and that process is the root process of the zone.  In general, all processes in a non-global zone are descendants of this init process.  The inheritable privilege set of init determines the effective privilege set of processes in that zone.  Note that a process is constrained by the set of privileges assigned to it when it was created, so by limiting the rights of init you limit the rights of any process created in that zone later.  Not even root can enable additional privileges in the zone.  In addition, standard system directories are mounted read-only (/lib, /platform, /usr, and /sbin), and the kernel is locked such that modules can’t be loaded or unloaded.  Not even root can alter this.

The administrator can enable system-wide privilege debugging by setting the system variable priv__debug = 1 in the global zone’s /etc/system file.

global# zlogin localzone
localzone# ls -l /tmp
total 8
drwxr-xr-x  2 root root   69 Apr 19 22:11 testdir
localzone# ppriv -D -e unlink /tmp/testdir
unlink[1245]: missing privilege "sys_linkdir"
              (euid = 0, syscall = 10)
              needed at tmp_remove+0x6e
unlink: Not owner
localzone# ppriv -D -e rmdir /tmp/testdir
localzone# ls -l /tmp
total 0

Docker Containers

Besides FreeBSD jails and Solaris containers, other container systems are available.  For Linux, you can install LXC (Linux containers).  See LXC at IBM DeveloperWorks for Linux for more information.  While other container technologies exist, undoubtedly Docker is the most popular one today (2017).

Docker is a tool for building and deploying applications by packaging them into containers.  A container can hold pretty much any software component along with all its dependencies (executables, libraries, configuration files etc.), and execute it in a guaranteed and repeatable runtime environment.  This makes it very easy to build your app once and deploy it anywhere: on your laptop for testing, then on different servers for live deployment etc.  These containers can be run from Linux systems, or from other OSes by using a small (25 MiB) wrapper for the container (essentially, a Linux VM).  Currently (2017), Docker is popular technology.

A Docker Image is an archive file (“filesystem bundle”) containing all the filesystem objects and meta-data needed to run a container.  There is a public repository of ready-to-use Docker images, and you can also create and use private repositories.

Docker images are built from simple recipe files called dockerfiles.  The command docker has many sub-commands, all that is needed to create, download, start/stop, and manage image files and containers.  Usually, the connection of the container to the outside world is handed by a number of daemons.

Docker itself does not provide any isolation.  Rather, it uses native Linux isolation features including the kernel’s firewall code, seccomp (to limit what system calls can be made from within the container), cgroups, namespaces, and ulimit.  All these low-level technologies can be managed via command line tools, but the complexity dissuades many from trying; those who do try often get something wrong in the configuration.  Experts at Docker have done all that work for you, with sensible and secure defaults that can be over-ridden on the command line.

Docker provides a similar level of isolation to that of FreeBSD jails and Solaris containers, but due to being provided by cgroups and namespaces, Linux containers can do more than isolate application processes.  (For example, Linux allows you to share namespaces between containers, so you can run Wireshark or strace in one container to see what’s happening in another.)

Managing multiple containers (a container cluster, or farm) requires orchestration software to deploy, load-balance, and manage containers in a cluster.  For example, a web server container may depend on a MariaDB container, and others.  If a container dies, another should probably be started, and the monitoring system updated.  If load increases, additional (identical) containers may be started and load-balanced.  Balancing the load on the physical servers requires the ability to move containers from one physical server to another.

This means clients wanting to contact a container-provided service must have some way to find it on your network; they need to determine the IP address and port number of the service.  This task is known as service discovery.  This task is harder than it may sound.  Some popular FOSS software that supports service discovery includes Apache Zookeeper, Doozer, Consul, Etcd, and Kubernetes (uses Etcd).

Some examples of orchestration software include Kubernetes (developed by Google) and fleet (part of CoreOS, a minimal distro designed just to run containers).  (Docker had developed Swarm for this, but now supports Kubernetes.)

Docker has proven so popular that other OSes want “in” (Mac and Windows).  A consortium was formed, the Open Container Initiative (OCI).  Docker has donated much of its technology to the OCI.  The future may include Windows and Mac native OCI images that run native binaries, thus avoiding Linux-only containers and requiring a Linux VM be running.

Virtual Machines and Emulators

It is known that single purpose systems can be made more secure than multiple purpose ones.  In the past, this has meant running each service on separate, dedicated-purpose hosts.  This was a very reasonable choice since server hardware was not capable of running many services well, and separate hardware was needed for performance reasons anyway.

Today, the economics have shifted again (as it always does eventually), and individual hosts are very capable of running several services at once.  However, this raises questions of security.  A vulnerability in any one service could lead to a compromise in all the other services running on that same host.  The normal isolation provided to separate processes running under a single kernel, isn’t enough to prevent many types of attacks (e.g., process A reading a file modified by process B).

Virtualization (a.k.a. para-virtualization, a.k.a. V12N) is a technique that can be used to isolate services (a set of processes) running on the same hardware from each other.  Each service runs in a separate instance of the operating system, or is otherwise isolated from other services.  This provides the same level of protection as provided by separate hosts, each running just one service.  Each OS “sees” different disks and hardware, and can’t affect one another anymore than they could when running on different hosts.

Virtualization is a popular solution for “big iron” (i.e., larger computers and mainframes), development work, testing, and for clusters, grids, and clouds.  Virtualization is also a general technique with many uses (e.g., virtual memory).  Sometimes it is used to permit running applications designed for a different OS or even different hardware (e.g., wine).  Application level virtualization is also common, including Microsoft’s CLI and Sun’s JVM.

Besides enhanced security (via increased isolation) and development support, virtualization can be used for other interesting purposes:

·       Students can practice networking by connections several virtual hosts in one or more virtual networks.  Such setups can be used to practice security as well; at worst, you end up corrupting a virtual hard disk.  You may not even need a backup copy of it, as some technologies permit read-only virtual machines (with any virtual disk changes saved in a separate file).  When you reboot the guest OS, you are back to the vanilla system!  The educational uses are endless.

·       Another use is versioning.  You can often clone a “guest” or virtual OS easily from a parent.  Then you upgrade one with new database, or version two of your website, or whatever.  Repeat for each major update to some service.  Then all old versions are available, and customers can migrate slowly from the old to the new.

·       Related to versioning is upgrading/patching with zero down time.  If the kernel is updated or certain other parts of the system (such as the glibc DLL, SELinux policy, or modules on a sealed kernel), a reboot is generally needed.  Worse, upgrading DLLs can cause running processes to become unstable.  It is therefore often best to upgrade on an off-line system, and then reboot it; but this can result in long down times.  But using virtualization, you can easily clone a system, update the clone off-line, and reboot the clone.  You can migrate clients with a flash-cut, or gradually via a load balancer to the new VM, and take the old one off-line when no sessions remain connected to it (a rolling update or restart).

·       Customer support is easier when you can use virtualization to mimic a customer’s environment, reproducing problems and testing solutions.

·       Testing is much cheaper (and easier) with virtualization.  (Now you have no excuse for not testing patches!)

·       Software configuration is made simpler, since the actual hardware used doesn’t matter.  You only need a single configuration for the virtual machine.

·       Demonstration software is reliable when run on a virtual machine, and easier to run without embarrassment at remote locations.

·       If your application servers run on virtual hosts you can easily rent cloud computing resources to temporarily expand your capacity (when you launch a new marketing campaign, for example).

Virtualization software is generally designed for a specific CPU architecture.  The i386 family of CPUs was not designed for this purpose; many x86 emulators have difficulty with at least some OSs and/or applications.  However, today’s (post-2006) CPUs do provide features to support virtualization, increasing both performance and security.

Intel calls their virtualization technology VT (or Vanderpool) and CPUs with this enabled show the flag vmx.  AMD calls theirs Pacifica and CPUs with this enabled show the flag svm (secure virtual machine).  To see if your CPU has virtualization support, run this:

          grep -E '^flags.*(vmx|svm)' /proc/cpuinfo

Other CPU extensions for virtual machine security and performance are now (2011) becoming common.

Intel’s VT needs activation in the BIOS before applications can make use of it.  Most systems disable this by default but make an option available to activate it.

Virtualization Techniques

Each VM supports some virtual devices, some (i.e., client versions) support more devices than others (i.e., server versions).  However, all VMs support one or more virtual disks, stored as image files on the host OS.  These virtual disks can be either fixed or dynamic.  A dynamic virtual disk starts off using no space and grows as files are added.  A fixed disk is created at its full size.  Fixed disks are faster but use larger files.

All virtualization software allows for the saving of the VM state to a file or files on the host OS.  This is similar to hibernation.  The VM can later be restored from the saved state.  VMware has two options for this.  With non-persistent disks, any changes made are valid for the current session only; the next time the VM is restarted, it reverts to that saved state.  The other option is called snapshots, where you can save the current state during some session, and if desired you can later revert to that snapshot.

Other products use different techniques.  Xen allows for read-only disks, in which all changes are saved in a separate file.  You can delete that file to revert the state.

As mentioned above a VM can be cloned.  The original is called the parent.  The clones take less space since they only need record the difference between the parent and clone VMs.  An instructor can setup a parent and have the students clone it for assignments.  (Usually the clone will have different MAC addresses for its NICs.)

Network support has three different modes (not counting non-networked mode).  For each mode, there is a virtual hub that the virtual NICs (vNICs) connect to.  In local (or host-only) mode each guest VM and the host OS have vNICs connected to the “local” hub.  They can communicate with each other but not with the physical NIC and in this mode, there is no Internet access.  The SA will need to assign each vNIC an IP address in the same network.

In bridged mode, the virtual NICs as well as the physical NIC are connected to the “bridged” hub.  In addition, the Host OS gets a vNIC connected to this hub too, and this becomes the default NIC for the host OS.  The vNICs need IP addresses in the same network as the physical NIC.  Bridged mode VMs can directly communicate with each other and the Internet (and any other reachable networks).  Bridged mode causes heavy LAN traffic and problems due to delays when switching between VMs.

With shared mode, each vNIC connects to the “shared” hub.  This is exactly the same as local mode, except a virtual router with NAT connects the shared hub to the physical NIC.  External hosts only see the host OS’s IP address, not the guest VM IP addresses.  Most V12N products include a DHCP server with the virtual router to configure the vNICs automatically.

It is possible for some VMs to use one mode and others to use a different mode.  But since each mode uses a separate hub, only VMs in the same mode can directly communicate.  Note some products refer to the hubs as switches or bridges.

Other features that some products offer is NTP (sync guest OSs to host OS time), folders that can be shared among the VMs (think “NFS” or “Samba”), enhanced UI support: optimized video drivers, drag-and-drop between VMs, remote management console, integrated mouse and keyboard switching between VMs, etc.

Virtualization (“V12N”) Products

The virtualization or emulator software that does all that is variously known as hypervisor, (virtual machine) monitor, machine emulator, and other such terms.  Hypervisors can sit atop of an OS, or not require any OS (you boot the hypervisor on the physical machine).  A hypervisor that requires an OS is called hosted.  VirtualBox is hosted.  The other type is called a bare-metal hypervisor.  VMware’s ESXi is one of these.

The virtual machines are often referred to as guests or domains.  Some technologies require applications or OS drivers (or whole operating systems) to be made compatible.  Examples include Denali and Xen.  With hardware CPU virtualization as provided by Intel VT and AMD Pacifica technology, the ability to run an unmodified guest OS kernel is available in many V12N products.  While no porting of the OS is required, some additional driver support may be necessary, such as for Xen.  Note that stopping the hypervisor (say by rebooting the host OS) has the same effect as turning off the power to each virtual machine.  You need to shutdown guest OSs first!

Usually you set up virtual machine instances with virtual hardware: disks, NICs, etc.  No two instances directly share anything.  Each sees its own disks, RAM, and NICs.  So no matter what sad fate befalls one service in one instance, the remaining services are unaffected.

Other technologies can run unmodified applications or guest operating systems.  To simplify PC and workstation support (with their many devices) some technologies both virtualize some things and emulate others.  VMware Client does this, requiring a hosted or underlying OS (“host OS”) on top of which runs VMware.  VMware Server (vSphere) enables one host to appear as a pool of secure (i.e., isolated) virtual servers which can run several different operating systems.  Unlike the client version, VMware Server does not need a host operating system (a bare-metal hypervisor).  for hosted hypervisors, the (initially booted) host OS used to create and manage other virtual machines is often referred to as domain 0 or the host OS.

Often the host OS needs virtualization support built into its kernel.  This is in addition to the hypervisor software (usually runs as a daemon).  Various utilities allow you to create, start, stop, monitor, and manage the hypervisor and the virtual machine instances.

Hypervisor-aware OSs perform better than unmodified OSs that require hardware emulation.  This is due to the CPU context switching to ring 0 and the extra code needed by the hypervisor to trap selective kernel mode access.

Some virtualization technologies emulate the hardware with software (called machine emulators).  Examples include QEMU (if not “accelerated”), Virtual PC, Bochs, and the Hercules emulator.

Others emulators only emulate a few critical components and allow the guest OS instance to access the rest of the hardware directly, saving and restoring the machine state when switching between instances.  Examples of this type include KQEMU (QEMU with acceleration), KVM (the Kernel-based Virtual Machine), Win4Lin, and VMware (both client and server versions; see also the free VMware Player).  Sun (was Innotek) VirtualBox supports everything, including VMware DOMs.

Other types of virtualization are possible.  OS virtualization is the name sometimes used when a single OS manages isolation for sets of processes.  Solaris zones or containers and FreeBSD jails also fall into this category.  (There is a Linux product as well, Virtuozzo.)  Sun also supports branded zones that include ABI support for foreign OSs (Linux and maybe Windows).