Analytics/Archive/Infrastructure/Meetings/SecurityReview

= Cluster Security =

Public Surfaces

 * ACL as outlined in the RT ticket.
 * Cross-datacenter connectivity (esams) requires a solution (public IP, or perhaps the existing bridge?)

Services and Access Needs
Many internal service dashboards and control panels:


 * Hue: HDFS web access; Hadoop job scheduling (via Oozie); Hive query dashboard (Beeswax)
 * Provides LDAP integration
 * We had wired it to use Labs LDAP and demand a specific project/group
 * Can control HDFS file permissions, allowing segregation of NDA/non-NDA data access, private data import, etc
 * Can control privileges within the dashboard to granularly restrict access to particular services,
 * Limited control can be exerted on resource use
 * Dashboard access needed by analysts need access for job monitoring/control, and data access
 * Hadoop Admin pages
 * NameNode: Provides HDFS logs and system health overview. Cluster administrator access only.
 * JobTracker, DataNode: provides logs and debugging output for Hadoop jobs. Access needed by analysts to debug.
 * Storm's Nimbus: storm job monitoring and scheduling. Cluster administrator access only.
 * Graphite: Application and host monitoring for the cluster. Cluster administrator access only.

= Data Retention =

What Potentially Private or Sensitive Data is there?

 * IP addresses
 * Browsing activity (including edits) correlated with Usernames & IPs
 * Data leakage via search queries / referrers
 * Low-entropy Data:
 * Precise geographic information about all activity
 * Login/Logout by IP across all web properties

Raw Logs

 * First persistantly stored in Kafka brokers; buffer window is 7 (?) days, and automatically deleted afterward
 * ETL anonymization pipeline sees raw data, removes IPs (salted hash) after geo lookup
 * Raw archival logs currently exist back to 2011 containing unsanitized IPs
 * wikistats processes all of time at once, and frequently has errors that require recomputation

Policies

 * Never publish raw logs or unaggregated datasets