Initial check-in for release

LICENSE

@@ -0,0 +1,11 @@
+Copyright 2021 The Pennsylvania State University
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,78 @@
+# Azure Usage Detail Export Function #
+A simple function to pull usage details out of an Azure enrollment and put them somewhere more useful. 
+
+## Prerequisites ##
+1. Google Cloud Project (BigQuery, Secrets Manager, Cloud Scheduler, Cloud Functions)
+2. Azure AD Service Principal/App Registration
+3. Enterprise Enrollment Access
+
+## Setup ##
+---
+Assign the Enrollment Reader role in your enrollment to the service principal. Steps to complete this can be found [here](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/assign-roles-azure-service-principals)  The guid representing the roles remains consistent across enrollments, only the billing profile id, aka. enrollment number, will change.
+*To successfully complete the role assignment, it must be the Object Id of the App Registration, not the Application Id.*
+
+
+A helper script is included to assist with creating the BigQuery table that will receive the data. The table is date partitioned to reduce query sizes and improve query performance. A view is also provided to simplify queries related to general billing questions eg. how much was spent on reserved instances's this year. The view will provide much the same information as the EA portal, but in a much more queryable fashion. 
+Clone this repository to a cloud shell, and install the python required modules.
+
+>`pip install -r requirements.txt` 
+
+Rename config.sample to config.py and edit the values to match the App Id/Client Id, and Tenant. Optionally you may change the Dataset and Table names to more appropriate values.  
+Use the gcloud command line to confirm or change the current project for your session.
+
+>`gcloud config get-value project`
+>
+>`gcloud config set project [projectId]` if needed to change projects  
+
+Run the table creation script. The script is idempotent and will not cause issues in re-applying.  
+
+> `python table.py` 
+
+Create Service Account
+>```
+> gcloud iam service-accounts create [service-account-name] \
+>   --display-name="Azure Usage Detail Export Function Service Account"
+>```
+
+Store the Azure client secret in Secrets Manger and assign Accessor Role to Service Account
+> `printf [ClientSecretValue] | gcloud secrets create [ClientSecretName] --data-file=-`
+
+>```
+>gcloud secrets add-iam-policy-binding [ClientSecret] \
+>  --member='serviceAccount:[service-account-name]@[projectId].iam.gserviceaccount.com' \
+>  --role='roles/secretmanager.secretAccessor'
+>```
+
+Edit cloudbuild.yaml To set the secret reference and the service account 
+>  \- --set-secrets=CLIENT_SECRET_KEY=[ClientSecretName]:latest
+>
+>  \- --service-account=[service-account-name]@[projectId].iam.gserviceaccount.com
+
+
+Deploy the function using Cloud Build  
+>  `gcloud builds submit --config=cloudbuild.yaml .`
+
+Assign Service Account the Data Editor role to the table
+> ```
+>bq add-iam-policy-binding \
+>   --member='serviceAccount:[service-account-name]@[projectId].iam.gserviceaccount.com' \
+>   --role='roles/bigquery.dataEditor' \
+>   [projectId]:[dataset].[table]
+>```
+
+Define the schedule for execution. The schedule can be repeated for each enrollment you have changing the enrollment value in the message body. The schedule exampled here is for 08:00 AM on the fifth day of each month.
+> `gcloud scheduler jobs create http [name] --location=us-central1 --schedule="0 8 5 * *" --message-body="{"enrollment":[EnrollmentNumber], "load_type":"monthly"}"  --uri=[httpsTrigger:url]` 
+
+## Optional Next Steps ##
+1. Commit changes to git and push to a new repository. Configure a Cloud Build trigger and deployment pipeline.
+2. Repeat the scheduled execution for additional enrollments. 
+
+## Known Limitations ## 
+### Azure data accuracy ###
+In testing the option to load data on a daily or weekly basis proved unstable. While data was brought in it was either duplicative, or missing entries compared to the EA portal or Cost Management in the standard Azure portal. This has been found up to a few days after the end of the month. Data retrieved on the 5th day for the previous month has been found to be accurate.  
+
+### Tag data ###
+Tags are currently dropped from the data set. While feasible to have them in the BigQuery data set, they need to be sanitized to ensure the keys and values are allowable. 
+
+## Contributing ##
+Contributions are welcome, especially for known issues, bugs, or to improve documentation. Please understand that this code is sufficient for our needs, it does not mean that is perfect, the correct way and it is far from the only way to collect this information. Sending an email will not guarantee a response, please use the issues functionality if you have problems.  

cloudbuild.yaml

@@ -0,0 +1,18 @@
+steps:
+- name: 'gcr.io/cloud-builders/gcloud'
+  waitFor: ['-']
+  id: azure-usage
+  args:
+  - beta
+  - functions
+  - deploy
+  - azure-usage-daily-export
+  - --region=us-central1
+  - --runtime=python37
+  - --memory=256MB
+  - --source=.
+  - --trigger-http
+  - --timeout=540
+  - --entry-point=azure_daily_billing_export
+  - --set-secrets=CLIENT_SECRET_KEY=
+  - --service-account=

config.sample

@@ -0,0 +1,13 @@
+import os
+
+config = dict()
+
+config['tenant'] = 'be5184f4-7ed8-4cd3-b3ec-abb1fe3fbe78'
+config['client_id'] = '6150e7fc-39c0-443b-a291-d59c7655ab51'
+# List of Azure Enrollment
+config['dataset']="CSP_Billing"
+config['table_name'] = "Azure_Daily_Export"
+
+# GCP Secret manager key refering to the AAD Client secret 
+if "CLIENT_SECRET_KEY" in os.environ:
+    config['CLIENT_SECRET_KEY'] = os.environ["CLIENT_SECRET_KEY"]

main.py

@@ -0,0 +1,214 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# =============================================================================
+# Created By  : Gabriel Geise
+# Created Date: Tue Nov 12 12:06:00 EST 2021
+# =============================================================================
+
+'''
+Cloud function triggered by HTTPS call with two parmeters in th body (enrollment #, and load_type). Load types accepted are daily, and monthly. 
+Daily loads filter for data from yesterday, monthly collects from the previous billing cycle.
+Data is extracted from the Azure Consumption API via a service principal. See https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/assign-roles-azure-service-principals
+for details on assigning a service principal to the enrollment.
+The data is then inserted into a Bigquery table for later querying. 
+'''
+import json
+import datetime
+import os
+from decimal import Decimal
+from config import config
+from google.cloud import bigquery
+
+bq_client = bigquery.Client(project="up-eit-ce-production")
+table_ref = bq_client.dataset(config['dataset']).table(config['table_name'])
+token = ""
+
+today = datetime.date.today()
+yesterday = today - datetime.timedelta(days=1)
+first = today.replace(day=1)
+lastMonth = first - datetime.timedelta(days=1)
+billing_start = lastMonth.replace(day=1)
+
+
+## Helper function to encode datetime objects
+def default(o):
+    if isinstance(o, (datetime.date, datetime.datetime)):
+        return o.isoformat()
+
+########################################
+# CloudFunction
+# Triggered via Cloud Scheduler 
+# Queries Azure Consumption API for detailed usage data
+########################################
+def azure_daily_billing_export (request):
+
+    from azure.mgmt.consumption import ConsumptionManagementClient
+    from azure.identity import ClientSecretCredential
+
+    request_json = request.get_json()
+
+    scope="/providers/Microsoft.Billing/billingAccounts/"
+
+    usage_json = []
+    print(request_json)
+    enrollment = request_json.get('enrollment')
+    load_type = request_json.get('load_type')
+    path = scope + str(enrollment) + "/providers/Microsoft.Billing/billingPeriods/"
+
+    if load_type == "daily" :
+        path += first.strftime("%Y%m")
+        usages = gather_usage_details(path)
+        billing_period=first.strftime("%Y-%m-%d")
+        usages = list(filter(lambda x: (x.date.date() == yesterday),usages))
+
+    elif load_type == "monthly":
+        path += billing_start.strftime("%Y%m")
+        usages = gather_usage_details(path)
+        billing_period=billing_start.strftime("%Y-%m-%d")
+
+    elif load_type == "this_month":
+        path += first.strftime("%Y%m")
+        usages = gather_usage_details(path)
+        billing_period=first.strftime("%Y-%m-%d")
+
+    else:
+        raise ValueError('Requested load type ('+load_type+') not valid')
+
+    print(billing_period)
+    usage_json.extend(map(lambda x: dict(
+        id=x.id,
+        name=x.name,
+        type=str(x.type),
+        kind=x.kind,
+        billing_account_id=x.billing_account_id,
+        billing_account_name=x.billing_account_name,
+        billing_period=billing_period,
+        billing_period_start_date=x.billing_period_start_date.isoformat(),
+        billing_period_end_date=x.billing_period_end_date.isoformat(),
+        billing_profile_id=x.billing_profile_id,
+        billing_profile_name=x.billing_profile_name,
+        account_owner_id=x.account_owner_id,
+        account_name=x.account_name,
+        subscription_id=x.subscription_id,
+        subscription_name=x.subscription_name,
+        date=x.date.isoformat(),
+        product=x.product,
+        part_number=x.part_number,
+        meter_id=x.meter_id,
+        meter_details=x.meter_details,
+        quantity=x.quantity,
+        effective_price=x.effective_price,
+        cost=x.cost,
+        unit_price=x.unit_price,
+        billing_currency=x.billing_currency,
+        resource_location=x.resource_location,
+        consumed_service=x.consumed_service,
+        resource_id=x.resource_id,
+        resource_name=x.resource_name,
+        service_info1=x.service_info1,
+        service_info2=x.service_info2,
+        additional_info=x.additional_info,
+        invoice_section=x.invoice_section,
+        cost_center=x.cost_center,
+        resource_group=x.resource_group,
+        reservation_id=x.reservation_id,
+        reservation_name=x.reservation_name,
+        product_order_id=x.product_order_id,
+        offer_id=x.offer_id,
+        is_azure_credit_eligible=x.is_azure_credit_eligible,
+        term=int(x.term) if x.term else None,
+        publisher_name=x.publisher_name,
+        publisher_type=x.publisher_type,
+        plan_name=x.plan_name,
+        charge_type=x.charge_type,
+        frequency=x.frequency
+        ),usages))
+    if load_type == "monthly" or load_type == "this_month":
+        purge_daily(billing_period, enrollment)
+    insert_to_bq(usage_json)
+
+
+def gather_usage_details(path):
+    from azure.mgmt.consumption import ConsumptionManagementClient
+    from azure.identity import ClientSecretCredential
+    cscredential = ClientSecretCredential(config['tenant'],config['client_id'], config['CLIENT_SECRET_KEY'])
+    consumption_client = ConsumptionManagementClient(
+        credential=cscredential,
+        subscription_id=''
+    )
+    print(path)
+
+    return consumption_client.usage_details.list(path)
+
+def insert_to_bq(data):
+    schema=[
+        bigquery.SchemaField("charge_type", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("plan_name", "STRING", "NULLABLE"),
+        bigquery.SchemaField("publisher_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("resource_group", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("invoice_section", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("service_info2", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("frequency", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("service_info1", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("reservation_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("resource_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("consumed_service", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("reservation_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("additional_info", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("resource_location", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("is_azure_credit_eligible", "BOOLEAN", "NULLABLE"),	
+        bigquery.SchemaField("billing_currency", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("unit_price", "FLOAT", "NULLABLE"),	
+        bigquery.SchemaField("cost", "FLOAT", "NULLABLE"),	
+        bigquery.SchemaField("effective_price", "FLOAT", "NULLABLE"),	
+        bigquery.SchemaField("quantity", "FLOAT", "NULLABLE"),	
+        bigquery.SchemaField("meter_details", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("product_order_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("cost_center", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("kind", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("meter_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("date", "TIMESTAMP", "NULLABLE", "bq-datetime"),	
+        bigquery.SchemaField("product", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("part_number", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("publisher_type", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("subscription_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("account_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("billing_period_end_date", "TIMESTAMP", "NULLABLE", "bq-datetime"),	
+        bigquery.SchemaField("offer_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("billing_account_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("account_owner_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("billing_profile_id", "INTEGER", "NULLABLE"),	
+        bigquery.SchemaField("subscription_id", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("resource_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("billing_profile_name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("name", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("billing_account_id", "INTEGER", "NULLABLE"),	
+        bigquery.SchemaField("billing_period_start_date", "TIMESTAMP", "NULLABLE", "bq-datetime"),
+        bigquery.SchemaField("billing_period", "DATE","NULLABLE"),
+        bigquery.SchemaField("type", "STRING", "NULLABLE"),	
+        bigquery.SchemaField("term", "INTEGER", "NULLABLE"),	
+        bigquery.SchemaField("id", "STRING", "NULLABLE")
+    ]
+
+    job_config = bigquery.LoadJobConfig()
+    job_config.write_disposition = bigquery.WriteDisposition.WRITE_APPEND
+    job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
+    job_config.schema = schema
+    load_job = bq_client.load_table_from_json(
+        data, table_ref, job_config=job_config
+    )  # API request
+
+    load_job.result()  # Waits for table load to complete.
+    print("{} Usage Table finished.".format("Azure"))
+
+def purge_daily(period, enrollment):
+    sql = "Delete FROM {dataset}.{table} where billing_period='{period}' and billing_profile_id={enrollment}".format(dataset=config['dataset'], table=config['table_name'], period=period, enrollment=enrollment)
+    query_job = bq_client.query(sql)  # Make an API request.
+    output = list(query_job)  # Wait for the job to complete.
+
+if __name__ == '__main__':
+    from unittest.mock import Mock
+    data = {"enrollment":64988483, "load_type":"monthly"}
+    req = Mock(get_json=Mock(return_value=data), args=data)
+
+    azure_daily_billing_export(req)

requirements.txt

@@ -0,0 +1,4 @@
+google-cloud-bigquery
+azure-identity
+azure-mgmt-consumption==8.0.0
+google-auth