Power BI + SharePoint Integration Is Ending Here’s Your PowerShell Script

With the typical heads-up, Microsoft announced that a somewhat underutilized feature in SharePoint Online is being phased out. The ability to create a Power BI report directly from a SharePoint list or library using the visualize this list feature will soon be a thing of the past. From this blog post, they provide an HTTP GET call to dump all of the reports, but not much else. What I’m providing in my script is more insight into what sites and lists have the reports bound to them. What my script is lacking is the detail needed to know who created the report or owns it. If you know how to capture this, please let me know, and I’ll update the script.

For this script to run, you will need:
PnP PowerShell
An Azure App Registration
Ability to access your tenant using az login
– It’s recommended that you use an account with Fabric administrator rights, tenant admin rights, or another level of access that can see all-thing-PowerBI.
If you are unable to utilize an App Reg, I provided another script below this one that will output the same basic detail, minus the target list display name.

In your PowerShell terminal, log in using AZ Login, then run the script.

#region Configuration UPDATE THE CONFIG VALUES
$config = @{
    ApiEndpoint = "https://api.powerbi.com/v1.0/myorg/admin/groups?`$filter=endswith(name,'SPList')&`$expand=reports,datasets,users&`$top=5000"
    OutputFile = "PowerBI_SPList_Workspaces_WithDatasources_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv"

    ClientId = "UPDATE THIS"
    Thumbprint = "UPDATE THIS"
    Tenant = "YourTenant" #contoso
}

$listDisplayNameCache = @{}
#endregion

#region Helper Functions
function Get-ListGuidFromEmail {
    param($emailAddress)
    if ($emailAddress -match "SPList-([a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12})") {
        return $matches[1]
    }
    return $null
}

function Get-SharePointListDisplayName {
    param([string]$siteUrl, [string]$listGuid)
    
    try {
        Connect-PnPOnline -Url $siteUrl -ClientId $config.ClientId -Thumbprint $config.Thumbprint -Tenant "$($config.Tenant).onmicrosoft.com" -ErrorAction Stop
        $list = Get-PnPList -Identity $listGuid -ErrorAction Stop
        return $list.Title
    }
    catch {
        Write-Host "      ⚠️  Could not retrieve list name for GUID $listGuid : $($_.Exception.Message)" -ForegroundColor Yellow
        return $null
    }
}

function Get-ListInfoFromWorkspace {
    param($workspace, $siteUrl)
    
    $listGuid = $null
    $listDisplayName = $null
    
    foreach ($user in $workspace.users) {
        $extractedGuid = Get-ListGuidFromEmail -emailAddress $user.emailAddress
        if ($extractedGuid) {
            $listGuid = $extractedGuid
            Write-Host "      Found List GUID: $listGuid" -ForegroundColor Cyan
            
            $cacheKey = "$siteUrl|$listGuid"
            if ($listDisplayNameCache.ContainsKey($cacheKey)) {
                $listDisplayName = $listDisplayNameCache[$cacheKey]
                Write-Host "      Using cached: $listDisplayName" -ForegroundColor Gray
            }
            else {
                Write-Host "      Querying SharePoint..." -ForegroundColor Yellow
                $listDisplayName = Get-SharePointListDisplayName -siteUrl $siteUrl -listGuid $listGuid
                if ($listDisplayName) {
                    Write-Host "      ✅ Display Name: $listDisplayName" -ForegroundColor Green
                    $listDisplayNameCache[$cacheKey] = $listDisplayName
                }
            }
            break
        }
    }
    
    return @{
        Guid = $listGuid
        DisplayName = $listDisplayName
    }
}

function New-WorkspaceResult {
    param(
        $workspace,
        $report = $null,
        $datasetId = $null,
        $datasource = $null,
        $listInfo = $null,
        $errorStatus = $null
    )
    
    $usersList = ($workspace.users | ForEach-Object { "$($_.emailAddress) [$($_.groupUserAccessRight)]" }) -join "; "
    
    return [PSCustomObject]@{
        WorkspaceName = $workspace.name
        WorkspaceId = $workspace.id
        WorkspaceType = $workspace.type
        WorkspaceState = $workspace.state
        ReportName = $report.name ?? "No Reports"
        ReportId = $report.id
        ReportWebUrl = $report.webUrl
        DatasetId = $datasetId
        DatasourceType = $datasource.datasourceType ?? $errorStatus
        DatasourceId = $datasource.datasourceId
        SharePointSiteUrl = $datasource.connectionDetails.url
        SharePointListGuid = $listInfo.Guid
        SharePointListDisplayName = $listInfo.DisplayName
        ConnectionString = $datasource.connectionDetails.url
        GatewayId = $datasource.gatewayId
        UserCount = $workspace.users.Count
        Users = $usersList
        QueryDate = Get-Date
    }
}
#endregion

#region Authentication
Write-Host "=== POWER BI WORKSPACE & DATASOURCE QUERY ===" -ForegroundColor Yellow
Write-Host "API Endpoint: $($config.ApiEndpoint)" -ForegroundColor Cyan
Write-Host "Output File: $($config.OutputFile)" -ForegroundColor Cyan

try {
    Write-Host "`nGetting access token..." -ForegroundColor Yellow
    $token = az account get-access-token --resource https://analysis.windows.net/powerbi/api --query accessToken -o tsv
    if (!$token) { throw "Failed to get access token" }
    Write-Host "✅ Token retrieved" -ForegroundColor Green
}
catch {
    Write-Host "❌ Failed to authenticate: $_" -ForegroundColor Red
    Write-Host "Please run 'az login' first" -ForegroundColor Yellow
    exit 1
}
#endregion

#region Query Power BI Workspaces
try {
    Write-Host "`nQuerying Power BI Admin API..." -ForegroundColor Yellow
    $headers = @{ "Authorization" = "Bearer $token"; "Content-Type" = "application/json" }
    $response = Invoke-RestMethod -Uri $config.ApiEndpoint -Headers $headers -Method Get
    $workspaces = $response.value
    
    Write-Host "✅ Found $($workspaces.Count) workspace(s)" -ForegroundColor Green
    if ($workspaces.Count -eq 0) { exit 0 }
}
catch {
    Write-Host "❌ API call failed: $_" -ForegroundColor Red
    exit 1
}
#endregion

#region Process Workspaces
Write-Host "`n=== PROCESSING WORKSPACES & DATASOURCES ===" -ForegroundColor Yellow
$allResults = @()
$counter = 1

foreach ($workspace in $workspaces) {
    Write-Host "`n--- Workspace $counter of $($workspaces.Count) ---" -ForegroundColor Cyan
    Write-Host "Name: $($workspace.name)" -ForegroundColor White
    
    $datasetIds = $workspace.reports | Where-Object { $_.datasetId } | Select-Object -ExpandProperty datasetId -Unique
    
    if (-not $datasetIds) {
        Write-Host "No datasets found" -ForegroundColor Yellow
        $allResults += New-WorkspaceResult -workspace $workspace
        $counter++
        continue
    }
    
    Write-Host "Datasets: $($datasetIds.Count)" -ForegroundColor Green
    
    foreach ($datasetId in $datasetIds) {
        Write-Host "`n  Dataset: $datasetId" -ForegroundColor Yellow
        
        try {
            $datasourceUrl = "https://api.powerbi.com/v1.0/myorg/admin/datasets/$datasetId/datasources"
            $datasourcesResponse = Invoke-RestMethod -Uri $datasourceUrl -Headers $headers -Method Get
            $datasources = $datasourcesResponse.value
            
            if (-not $datasources -or $datasources.Count -eq 0) {
                Write-Host "  ⚠️  No datasources found" -ForegroundColor Yellow
                continue
            }
            
            Write-Host "  ✅ Found $($datasources.Count) datasource(s)" -ForegroundColor Green
            
            foreach ($datasource in $datasources) {
                Write-Host "    Type: $($datasource.datasourceType)" -ForegroundColor Gray
                
                $siteUrl = $datasource.connectionDetails.url
                $listInfo = Get-ListInfoFromWorkspace -workspace $workspace -siteUrl $siteUrl
                $reportsUsingDataset = $workspace.reports | Where-Object { $_.datasetId -eq $datasetId }
                
                foreach ($report in $reportsUsingDataset) {
                    $allResults += New-WorkspaceResult -workspace $workspace -report $report -datasetId $datasetId -datasource $datasource -listInfo $listInfo
                }
            }
        }
        catch {
            Write-Host "  ⚠️  Error: $($_.Exception.Message)" -ForegroundColor Yellow
            $reportsUsingDataset = $workspace.reports | Where-Object { $_.datasetId -eq $datasetId }
            foreach ($report in $reportsUsingDataset) {
                $allResults += New-WorkspaceResult -workspace $workspace -report $report -datasetId $datasetId -errorStatus "Error retrieving"
            }
        }
        
        Start-Sleep -Milliseconds 100
    }
    
    $counter++
}
#endregion

#region Export and Summary
if ($allResults.Count -gt 0) {
    Write-Host "`n=== EXPORTING RESULTS ===" -ForegroundColor Yellow
    $allResults | Export-Csv -Path $config.OutputFile -NoTypeInformation
    Write-Host "✅ Exported to: $($config.OutputFile)" -ForegroundColor Green
    Write-Host "   Total records: $($allResults.Count)" -ForegroundColor Cyan
}

Write-Host "`n=== SUMMARY ===" -ForegroundColor Yellow
Write-Host "Total Workspaces: $($workspaces.Count)" -ForegroundColor Cyan
Write-Host "Total Records: $($allResults.Count)" -ForegroundColor Cyan

$uniqueSites = $allResults | Where-Object { $_.SharePointSiteUrl } | Select-Object -ExpandProperty SharePointSiteUrl -Unique
if ($uniqueSites) {
    Write-Host "`nUnique SharePoint Sites: $($uniqueSites.Count)" -ForegroundColor Cyan
    $uniqueSites | ForEach-Object { Write-Host "  - $_" -ForegroundColor Gray }
}

Write-Host "`n✅ Query completed successfully!" -ForegroundColor Green
#endregion


———- Other script ———————-

Here’s the same basic script, but without needing the Azure App Reg. You will need to grab the SharePointListGuid value, navigate to the corresponding site, open the site contents page, and then click on any list or library settings. In the URL, swap the existing value with the one from SharePointListGuid. This will display the name of the list to which the current report is bound.

Example:
https://taco.sharepoint.com/sites/Mas-Taco/_layouts/15/listedit.aspx?List=5ae2c019-75a5-495c-a5a9-beaf47b71346
You’ll want to swap the GUID after ?List=

#region Configuration
$config = @{
    ApiEndpoint = "https://api.powerbi.com/v1.0/myorg/admin/groups?`$filter=endswith(name,'SPList')&`$expand=reports,datasets,users&`$top=5000"
    OutputFile = "PowerBI_SPList_Workspaces_$(Get-Date -Format 'yyyyMMdd_HHmmss').csv"
}
#endregion

#region Helper Functions
function Get-ListGuidFromEmail {
    param($emailAddress)
    if ($emailAddress -match "SPList-([a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12})") {
        return $matches[1]
    }
    return $null
}

function New-WorkspaceResult {
    param(
        $workspace,
        $report = $null,
        $datasetId = $null,
        $datasource = $null,
        $listGuid = $null
    )
    
    $usersList = ($workspace.users | ForEach-Object { "$($_.emailAddress) [$($_.groupUserAccessRight)]" }) -join "; "
    
    return [PSCustomObject]@{
        WorkspaceName = $workspace.name
        WorkspaceId = $workspace.id
        WorkspaceType = $workspace.type
        WorkspaceState = $workspace.state
        ReportName = $report.name ?? "No Reports"
        ReportId = $report.id
        ReportWebUrl = $report.webUrl
        DatasetId = $datasetId
        DatasourceType = $datasource.datasourceType
        DatasourceId = $datasource.datasourceId
        SharePointSiteUrl = $datasource.connectionDetails.url
        SharePointListGuid = $listGuid
        ConnectionString = $datasource.connectionDetails.url
        GatewayId = $datasource.gatewayId
        UserCount = $workspace.users.Count
        Users = $usersList
        QueryDate = Get-Date
    }
}
#endregion

#region Authentication
Write-Host "=== POWER BI WORKSPACE QUERY (Simple Version) ===" -ForegroundColor Yellow
Write-Host "API Endpoint: $($config.ApiEndpoint)" -ForegroundColor Cyan
Write-Host "Output File: $($config.OutputFile)" -ForegroundColor Cyan

try {
    Write-Host "`nGetting access token..." -ForegroundColor Yellow
    $token = az account get-access-token --resource https://analysis.windows.net/powerbi/api --query accessToken -o tsv
    if (!$token) { throw "Failed to get access token" }
    Write-Host "✅ Token retrieved" -ForegroundColor Green
}
catch {
    Write-Host "❌ Failed to authenticate: $_" -ForegroundColor Red
    Write-Host "Please run 'az login' first" -ForegroundColor Yellow
    exit 1
}
#endregion

#region Query Power BI Workspaces
try {
    Write-Host "`nQuerying Power BI Admin API..." -ForegroundColor Yellow
    $headers = @{ "Authorization" = "Bearer $token"; "Content-Type" = "application/json" }
    $response = Invoke-RestMethod -Uri $config.ApiEndpoint -Headers $headers -Method Get
    $workspaces = $response.value
    
    Write-Host "✅ Found $($workspaces.Count) workspace(s)" -ForegroundColor Green
    if ($workspaces.Count -eq 0) { exit 0 }
}
catch {
    Write-Host "❌ API call failed: $_" -ForegroundColor Red
    exit 1
}
#endregion

#region Process Workspaces
Write-Host "`n=== PROCESSING WORKSPACES ===" -ForegroundColor Yellow
$allResults = @()
$counter = 1

foreach ($workspace in $workspaces) {
    Write-Host "`n--- Workspace $counter of $($workspaces.Count) ---" -ForegroundColor Cyan
    Write-Host "Name: $($workspace.name)" -ForegroundColor White
    
    # Extract list GUID from workspace users (no SharePoint connection needed)
    $listGuid = $null
    foreach ($user in $workspace.users) {
        $extractedGuid = Get-ListGuidFromEmail -emailAddress $user.emailAddress
        if ($extractedGuid) {
            $listGuid = $extractedGuid
            Write-Host "List GUID: $listGuid" -ForegroundColor Cyan
            break
        }
    }
    
    $datasetIds = $workspace.reports | Where-Object { $_.datasetId } | Select-Object -ExpandProperty datasetId -Unique
    
    if (-not $datasetIds) {
        Write-Host "No datasets found" -ForegroundColor Yellow
        $allResults += New-WorkspaceResult -workspace $workspace -listGuid $listGuid
        $counter++
        continue
    }
    
    Write-Host "Datasets: $($datasetIds.Count)" -ForegroundColor Green
    
    foreach ($datasetId in $datasetIds) {
        Write-Host "  Dataset: $datasetId" -ForegroundColor Yellow
        
        try {
            $datasourceUrl = "https://api.powerbi.com/v1.0/myorg/admin/datasets/$datasetId/datasources"
            $datasourcesResponse = Invoke-RestMethod -Uri $datasourceUrl -Headers $headers -Method Get
            $datasources = $datasourcesResponse.value
            
            if (-not $datasources -or $datasources.Count -eq 0) {
                Write-Host "  No datasources found" -ForegroundColor Yellow
                continue
            }
            
            Write-Host "  ✅ Found $($datasources.Count) datasource(s)" -ForegroundColor Green
            
            foreach ($datasource in $datasources) {
                $siteUrl = $datasource.connectionDetails.url
                Write-Host "    Site: $siteUrl" -ForegroundColor Gray
                
                $reportsUsingDataset = $workspace.reports | Where-Object { $_.datasetId -eq $datasetId }
                
                foreach ($report in $reportsUsingDataset) {
                    $allResults += New-WorkspaceResult -workspace $workspace -report $report -datasetId $datasetId -datasource $datasource -listGuid $listGuid
                }
            }
        }
        catch {
            Write-Host "  ⚠️  Error: $($_.Exception.Message)" -ForegroundColor Yellow
            $reportsUsingDataset = $workspace.reports | Where-Object { $_.datasetId -eq $datasetId }
            foreach ($report in $reportsUsingDataset) {
                $allResults += New-WorkspaceResult -workspace $workspace -report $report -datasetId $datasetId -listGuid $listGuid
            }
        }
        
        Start-Sleep -Milliseconds 100
    }
    
    $counter++
}
#endregion

#region Export and Summary
if ($allResults.Count -gt 0) {
    Write-Host "`n=== EXPORTING RESULTS ===" -ForegroundColor Yellow
    $allResults | Export-Csv -Path $config.OutputFile -NoTypeInformation
    Write-Host "✅ Exported to: $($config.OutputFile)" -ForegroundColor Green
    Write-Host "   Location: $(Get-Location)$($config.OutputFile)" -ForegroundColor Green
    Write-Host "   Total records: $($allResults.Count)" -ForegroundColor Cyan
}

Write-Host "`n=== SUMMARY ===" -ForegroundColor Yellow
Write-Host "Total Workspaces: $($workspaces.Count)" -ForegroundColor Cyan
Write-Host "Total Records: $($allResults.Count)" -ForegroundColor Cyan

# Show unique SharePoint sites found
$uniqueSites = $allResults | Where-Object { $_.SharePointSiteUrl } | Select-Object -ExpandProperty SharePointSiteUrl -Unique
if ($uniqueSites) {
    Write-Host "`nUnique SharePoint Sites: $($uniqueSites.Count)" -ForegroundColor Cyan
    $uniqueSites | ForEach-Object { Write-Host "  - $_" -ForegroundColor Gray }
}

# Show unique list GUIDs found
$uniqueListGuids = $allResults | Where-Object { $_.SharePointListGuid } | Select-Object -ExpandProperty SharePointListGuid -Unique
if ($uniqueListGuids) {
    Write-Host "`nUnique SharePoint List GUIDs: $($uniqueListGuids.Count)" -ForegroundColor Cyan
    $uniqueListGuids | ForEach-Object { Write-Host "  - $_" -ForegroundColor Gray }
}

Write-Host "`n✅ Query completed successfully!" -ForegroundColor Green
#endregion

From a target SharePoint list or library, locate the Integrate tab, click it, click Power BI, and from there, you should see your report name.

Get SharePoint Site ID Using Microsoft Graph

Using Microsoft Graph, how do you get the SharePoint site ID using the site path?

Endpoint being used: https://learn.microsoft.com/en-us/graph/api/site-getbypath?view=graph-rest-1.0

# Function to acquire an access token and return the token and its expiration time
function Get-GraphAccessToken {
    $tokenUrl = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
    $body = @{
        client_id     = $clientId
        scope         = "https://graph.microsoft.com/.default"
        client_secret = $clientSecret
        grant_type    = "client_credentials"
    }

    $tokenResponse = Invoke-RestMethod -Uri $tokenUrl -Method Post -Body $body -ContentType "application/x-www-form-urlencoded"
    $accessToken = $tokenResponse.access_token
    $expiresIn = $tokenResponse.expires_in
    $tokenExpiration = (Get-Date).AddSeconds($expiresIn - 300) # Refresh 5 minutes before expiration

    return $accessToken, $tokenExpiration
}
# Acquire the token
$accessToken, $tokenExpiration = Get-GraphAccessToken
$headers = @{
    Authorization  = "Bearer $accessToken"
    "Content-Type" = "application/json"
}

# Target SharePoint site
$siteUrl = "https://taco.sharepoint.com/sites/test-site" 

# Extract the hostname and site path
$uri = [System.Uri]::new($siteUrl)
$hostname = $uri.Host
$sitePath = $uri.AbsolutePath.TrimStart('/')

# Define the endpoint URL to get the SharePoint site ID
$graphSiteUrl = "https://graph.microsoft.com/v1.0/sites/${hostname}:/${sitePath}"

# Make the request to get the site ID
$siteResponse = Invoke-RestMethod -Uri $graphSiteUrl -Headers $headers -Method Get

# Extract the relevant part of the site ID
$siteIdParts = $siteResponse.id.Split(',')
$siteId = "$($siteIdParts[1]),$($siteIdParts[2])"

# Output the site ID
Write-Output "Site Collection ID: $($siteResponse.siteCollection.hostname)"
Write-Output "Site ID--------> $siteId"
Write-Output "Site Display Name: $($siteResponse.displayName)"
Write-Output "Site Web URL: $($siteResponse.webUrl)"

This will return a value like 2480e89d-303a-4f38-b4fe-27f824ff88ac,d605ce5c-f356-422a-84fe-1d7820bc9e6d , which represents the site collection ID and the site ID.

Site collection ID: 2480e89d-303a-4f38-b4fe-27f824ff88ac
Site ID: d605ce5c-f356-422a-84fe-1d7820bc9e6d

Use Python to upload a LARGE file to SharePoint

In this post, I will quickly show how to use the Office365-REST-Python-Client library to upload a large file to a SharePoint library.

For this to work, you will need a certificate, Azure App registration, and access to the target SharePoint site. I outlined all the necessary parts in this post: Modernizing Authentication in SharePoint Online Note: the linked post will output a .PFX cert, and the script below will need a .PEM cert. You can use this Python command to convert the cert:

from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography import x509
from cryptography.hazmat.primitives.serialization import pkcs12

# Load the PFX file
pfx_file = open('C:\\path_to_cert\\EXAMPLE.pfx', 'rb').read()  # replace with your pfx file path
(private_key, certificate, additional_certificates) = pkcs12.load_key_and_certificates(pfx_file, None, default_backend())

with open('NewCERT.pem', 'wb') as f:
    f.write(private_key.private_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PrivateFormat.TraditionalOpenSSL,
        encryption_algorithm=serialization.NoEncryption()
    ))
    f.write(certificate.public_bytes(serialization.Encoding.PEM))

# install this library if needed
# pip install cryptography

Ok, with that out of the way, you can use this script to upload to a SharePoint library. In the script, I’ve commented out the line that would be used to upload to a folder within a library.

import os
from office365.sharepoint.client_context import ClientContext

cert_credentials = {
    "tenant": "abcxyz-1234-4567-8910-0e3d638792fb",
    "client_id": "abcddd-4444-4444-cccc-123456789111",
    "thumbprint": "7D8D8DF7D8D2F4DF8DF45D4FD8FD48DF5D8D",
    "cert_path": "RestClient\\NewCERT.pem"
}
ctx = ClientContext("https://tacoranch.sharepoint.com/sites/python").with_client_certificate(**cert_credentials)
current_web = ctx.web.get().execute_query()
print("{0}".format(current_web.url))

filename = "LargeExcel.xlsx"
folder_path = "C:\\code\py"

def print_upload_progress(offset):
    # type: (int) -> None
    file_size = os.path.getsize(local_path)
    print(
        "Uploaded '{0}' bytes from '{1}'...[{2}%]".format(
            offset, file_size, round(offset / file_size * 100, 2)
        )
    )

#upload to a folder
#target_url = "Shared Documents/folderA/folderB"

target_url = "Shared Documents"
target_folder = ctx.web.get_folder_by_server_relative_url(target_url)
size_chunk = 1000000
local_path = os.path.join(folder_path, filename)
with open(local_path, "rb") as f:
    uploaded_file = target_folder.files.create_upload_session(
        f, size_chunk, print_upload_progress, filename
    ).execute_query()

print("File {0} has been uploaded successfully".format(uploaded_file.serverRelativeUrl))

If you receive an error stating you don’t have access, double-check that you’ve added the App Registration to the target SharePoint site permissions. Again, this is noted in the blog post linked at the being of this post.

Consider this a workaround until MS Graph is out of its latest beta and there’s more support for easily uploading to SharePoint.

What if you need to upload a file and set a column value? When working with SharePoint via the API, you must be mindful of the column names. The column name in the UI might not be the same as the internal name, so I will use the script above as my starting point and add the following script to the end. In this example, I’m setting two fields: ReportName and ReportDate.

#get the file that was just uploaded
file_item = uploaded_file.listItemAllFields

# Define a dictionary of field names and their new values
fields_to_update = {
    "ReportName": "My TPS Report",
    "ReportDate": datetime.datetime.now().isoformat(),
    # Add more fields here as needed
}

# Iterate over the dictionary and update each field
for field_name, new_value in fields_to_update.items():
    file_item.set_property(field_name, new_value)

# Commit the changes
file_item.update()
ctx.execute_query()

print("Report fields were updated")

How do you get a list of all the columns in a list or library? The script below will output all the column’s internal and display names.

from office365.sharepoint.client_context import ClientContext

cert_credentials = {
    "tenant": "abcxyz-1234-4567-8910-0e3d638792fb",
    "client_id": "abcddd-4444-4444-cccc-123456789111",
    "thumbprint": "7D8D8DF7D8D2F4DF8DF45D4FD8FD48DF5D8D",
    "cert_path": "RestClient\\NewCERT.pem"
}

ctx = ClientContext("https://tacoranch.sharepoint.com/sites/python").with_client_certificate(**cert_credentials)
current_web = ctx.web.get().execute_query()
print("{0}".format(current_web.url))

# Get the target list or library
list_or_library = ctx.web.lists.get_by_title('TPS-Reports')

# Load the fields
fields = list_or_library.fields.get().execute_query()

# Print the field names
for field in fields:
    print("Field internal name: {0}, Field display name: {1}".format(field.internal_name, field.title))



Modernizing Authentication in SharePoint Online

Starting a year or two ago, Microsoft announced it would stop supporting and/or blocking access to Azure Access Control Services (ACS) and the SharePoint Add-In model. This is important because ACS has been used for many years to grant app/script API access to a SharePoint site(s), and you likely have many sites where this has been used. Moving forward, Azure Access Control (AAC) will be used in place of ACS.

Historically, you would start the permissions journey by generating the client ID and secret at this endpoint:
_layouts/15/AppRegNew.aspx
From there, you grant the newly created identity access to a tenant, sites, lists, libraries, or a combination.
_layouts/15/appinv.aspx

With that out of the way, how do you wire up a new connection to SharePoint Online, allowing PowerShell, Python, script, or app access to the SharePoint API or the Microsoft Graph API?

Overview of what I’m doing:
Create a self-signed cert
Add the cert to your personal cert store and upload it to Azure, creating an App Registration.
Adjust permissions as needed.
Grant the App Registration access to a specific SharePoint site.
Use the newly created credentials to access the SharePoint site.

#create cert with a password
New-PnPAzureCertificate `
    -CommonName "Demo_SP_Azure_2024" `
    -OutPfx c:\code\Demo_SP_Azure_2024.pfx `
    -OutCert c:\code\Demo_SP_Azure_2024.cer `
    -CertificatePassword (ConvertTo-SecureString -String "Taco@Good" -AsPlainText -Force) `
    -ValidYears 1

#import the cert. for this to work, run as Admin.
Import-PfxCertificate `
    -Exportable `
    -CertStoreLocation Cert:\LocalMachine\My `
    -FilePath c:\code\Demo_SP_Azure_2024.pfx `
    -Password (ConvertTo-SecureString -String "Taco@Good" -AsPlainText -Force)

I highly suggest not skipping the ‘-interactive’ part for the next command. It will open a browser window where you must authenticate with an account with adequate permissions to create a new App Registration in Azure. The script’s most important thing to note is SharePointApplicationPermissions Site.Selected. Why is this important? This is extremely useful if you want to limit permissions to a single SharePoint site and not every site in the tenant.

Register-PnPAzureADApp `
   -ApplicationName Demo_SP_Azure_2024 `
   -Tenant tacoranch.onmicrosoft.com `
   -Store CurrentUser `
   -SharePointApplicationPermissions "Sites.Selected" `
   -Interactive

After that runs, the output will include the Client ID and Certificate Thumprint. Take note of it, or you can open Azure, navigate to App Registrations, and select all applications. In the left nav, click Certificates & secrets, where you’ll find the thumbprint; again, in the left nav, click Overview, and you’ll see the Application ID, aka Client ID.

Edit:
Ensure the account running the next set of commands is a Site Collection Administrator on the target site. If the account does not have access to the site, you will receive the error noted at the bottom of this post.

In the next two commands, you will connect to the SharePoint Admin site interactive, then grant the newly created App Registration write access to the target SharePoint site.

Connect-PnPOnline -Url "https://tacoranch-admin.sharepoint.com" `
    -Interactive

#grant the App Reg write access to the site
Grant-PnPAzureADAppSitePermission `
    -AppId 'a95ddafe-6eca-488a-b26a-dc62a64d9105' `
    -DisplayName 'Demo_SP_Azure_2024' `
    -Site 'https://tacoranch.sharepoint.com/sites/cert-demo' `
    -Permissions Write


Now that the App Registration can access the SharePoint site, how do you connect to it using the certificate and thumbprint?

Connect-PnPOnline `
    -Tenant "tacoranch.onmicrosoft.com" `
    -Url "https://tacoranch.sharepoint.com/sites/cert-demo" `
    -ClientId "a95ddafe-6eca-488a-b26a-dc62a64d9105" `
    -Thumbprint "5C5891197B54B9535D171D6D9DD7D6D351039C8E" 

Get-PnPList | Select-Object Title

Using the above commands, you can create a cert, register it in Azure, and grant access to a single SharePoint site.

I’ve included a copy of the full script here:
https://www.sharepointed.com/wp-content/uploads/2024/03/Azure-App-Reg-Cert-Demo.txt

Error(s) and fixes:
Error:
Grant-PnPAzureADAppSitePermission: {"error":{"code":"accessDenied","message":"Access denied","innerError":{"date":"2024-03-21T16:29:47","request-id":"","client-request-id":""}}}
Fix:
Ensure the account running this command Grant-PnPAzureADAppSitePermission , has access to the target SharePoint site.

Error:
Connect-PnPOnline: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS700027: The certificate with identifier used to sign the client assertion is not registered on application. [Reason - The key was not found., Thumbprint of key used by client: '6C5891197B54B9535D179D6D9DD7D6D351039D8Z', Please visit the Azure Portal, Graph Explorer or directly use MS
Graph to see configured keys for app Id 'eb7f9fbc-f4ee-4a48-a008-49d6bcdc6c40'. Review the documentation at https://docs.microsoft.com/en-us/graph/deployments to determine the corresponding service endpoint and https://docs.microsoft.com/en-us/graph/api/application-get?view=graph-rest-1.0&tabs=http to build a query request URL, such as 'https://graph.microsoft.com/beta/applications/aeb7f9fbc-f4ee-4a48-a008-49d6bcdc6c40']. Trace ID: 3e1e7ab3-60c0-4126-acb8-e2fdb6e28000 Correlation ID: 62ddc80b-aeb2-49c5-8c31-83a04e70bf6e Timestamp: 2024-04-01 12:21:59Z

Fix:
This one is easy; ensure you use the correct Client ID and thumbprint. You can get this from the app registration page in the Azure portal.

How to upload a large file to SharePoint using the Microsoft Graph API

What started as a simple question from a co-worker turned into a rabbit hole exploration session that lasted a bit longer than anticipated. ‘Hey, I need to upload a report to SharePoint using Python.’

In the past, I’ve used SharePoint Add-in permissions to create credentials allowing an external service, app, or script to write to a site, library, list, or all of the above. However, the environment I’m currently working in does not allow Add-in permissions, and Microsoft has been slowly depreciating the service for a long time.

As of today (March 18, 2024) this is the only way I could find to upload a large file to SharePoint. Using the MS Graph SDK, you can upload files smaller than 4mb, but that is useless in most cases.

For the script below, the following items are needed:
Azure App Registration:
Microsoft Graph application permissions:
Files.ReadWrite.All
Sites.ReadWrite.All
SharePoint site
SharePoint library (aka drive)
File to test with

import requests
import msal
import atexit
import os.path
import urllib.parse
import os

TENANT_ID = '19a6096e-3456-7890-abcd-19taco8cdedd'
CLIENT_ID = '0cd0453d-cdef-xyz1-1234-532burrito98'
CLIENT_SECRET  = '.i.need.tacos-and.queso'
SHAREPOINT_HOST_NAME = 'tacoranch.sharepoint.com'
SITE_NAME = 'python'
TARGET_LIBRARY = 'reports'
UPLOAD_FILE = 'C:\\code\\test files\\LargeExcel.xlsx'
UPLOAD_FILE_NAME = 'LargeExcel.xlsx'
UPLOAD_FILE_DESCRIPTION = 'A large excel file' #not required

AUTHORITY = 'https://login.microsoftonline.com/' + TENANT_ID
ENDPOINT = 'https://graph.microsoft.com/v1.0'

SCOPES = [
    'Files.ReadWrite.All',
    'Sites.ReadWrite.All'
]

cache = msal.SerializableTokenCache()

if os.path.exists('token_cache.bin'):
    cache.deserialize(open('token_cache.bin', 'r').read())

atexit.register(lambda: open('token_cache.bin', 'w').write(cache.serialize()) if cache.has_state_changed else None)

SCOPES = ["https://graph.microsoft.com/.default"]

app = msal.ConfidentialClientApplication(CLIENT_ID, authority=AUTHORITY, client_credential=CLIENT_SECRET, token_cache=cache)

result = None
result = app.acquire_token_silent(SCOPES, account=None)

drive_id = None

if result is None:
    result = app.acquire_token_for_client(SCOPES)

if 'access_token' in result:
    print('Token acquired')
else:
    print(result.get('error'))
    print(result.get('error_description'))
    print(result.get('correlation_id')) 

if 'access_token' in result:
    access_token = result['access_token']
    headers={'Authorization': 'Bearer ' + access_token}

    # get the site id
    result = requests.get(f'{ENDPOINT}/sites/{SHAREPOINT_HOST_NAME}:/sites/{SITE_NAME}', headers=headers)
    result.raise_for_status()
    site_info = result.json()
    site_id = site_info['id']

    # get the drive / library id
    result = requests.get(f'{ENDPOINT}/sites/{site_id}/drives', headers=headers)
    result.raise_for_status()
    drives_info = result.json()
    
    for drive in drives_info['value']:
        if drive['name'] == TARGET_LIBRARY:
            drive_id = drive['id']
            break

    if drive_id is None:
        print(f'No drive named "{TARGET_LIBRARY}" found')

    # upload a large file to
    file_url = urllib.parse.quote(UPLOAD_FILE_NAME)
    result = requests.post(
        f'{ENDPOINT}/drives/{drive_id}/root:/{file_url}:/createUploadSession',
        headers=headers,
        json={
            '@microsoft.graph.conflictBehavior': 'replace',
            'description': UPLOAD_FILE_DESCRIPTION,
            'fileSystemInfo': {'@odata.type': 'microsoft.graph.fileSystemInfo'},
            'name': UPLOAD_FILE_NAME
        }
    )

    result.raise_for_status()
    upload_session = result.json()
    upload_url = upload_session['uploadUrl']

    st = os.stat(UPLOAD_FILE)
    size = st.st_size
    CHUNK_SIZE = 10485760
    chunks = (size + CHUNK_SIZE - 1) // CHUNK_SIZE
    with open(UPLOAD_FILE, 'rb') as fd:
        start = 0
        for chunk_num in range(chunks):
            chunk = fd.read(CHUNK_SIZE)
            bytes_read = len(chunk)
            upload_range = f'bytes {start}-{start + bytes_read - 1}/{size}'
            print(f'chunk: {chunk_num} bytes read: {bytes_read} upload range: {upload_range}')
            result = requests.put(
                upload_url,
                headers={
                    'Content-Length': str(bytes_read),
                    'Content-Range': upload_range
                },
                data=chunk
            )
            result.raise_for_status()
            start += bytes_read

else:
    raise Exception('no access token')

In the script, I’m uploading the LargeExcel file to a library named reports in the Python site. It is important to note that the words drive and library are used interchangeably when working with MS Graph. If you see a script example that does not specify a target library but only uses root, it will write the files to the default Documents / Shared Documents library.

Big thank you to Keath Milligan for providing the foundation of the script.
https://gist.github.com/keathmilligan/590a981cc629a8ea9b7c3bb64bfcb417

How to Find Your Microsoft Forms Data: Locating the Linked Excel File

This started as a simple question: where is the backend Excel file for my group Forms form stored?

By default, a group form will save responses to an Excel file in the SharePoint site associated with the group. Within that site, the file is stored in the Documents, aka Shared Documents library.

Here is a quick way to track down the file:

With the form open, click on Response.

Click on Open in Excel.

Depending on how your SharePoint library is configured, the file will either download to your computer or open in the browser. Open the file and click on the name or click the down arrow next to it.

When the window opens, it will show exactly where the file is stored.

In this example, the file is stored in the Shared Documents library on the Testing site. Again, this example shows Shared Documents, but on the site, it’s actually named Documents.

STOP, I don’t see the window noted in the above screenshot! This more than likely means you are working with a personal form.

Where is the Excel file stored for personal forms? Not where you’d guess and not anywhere worthwhile. The file is more or less saved with the form and is inaccessible other than downloading it.
Personal Forms response data is now stored in the author’s OneDrive / SharePoint site.

What if I copy my personal form to a group? What will happen to the Excel file?
Don’t do this; just recreate the form from scratch. The copied form will retain the behavior of storing the file with the form, not in SharePoint.

How can I save form responses to a SharePoint list or Dataverse table? You would need to create a Flow to intercept the form response and then save it to the destination.

Will creating a Flow that saves form responses to another destination impact the form saving to Excel? No, the form will always use the backend Excel file as its data storage.

If I download a copy of the backend Excel file, will the downloaded copy be updated with new form submissions? No, the copy is disconnected from the source.

How do you find ALL the Flows that reference a SharePoint site or list?

I asked this question when I first started down the path of learning about Flow:
How do you find all the Flows running on or referencing a SharePoint list?

UPDATE / EDIT – READ THIS Part
Before you start on this, please ensure that your account or the account you are using to run the script has sufficient permissions to the target environment(s).

$oneFlow = Get-AdminFlow -FlowName "00000-ae95-4cab-96d8-0000000" -EnvironmentName "222222-4943-4068-8a2d-11111111"

$refResources = $oneFlow.Internal.properties.referencedResources
Write-Host $refResources



If you run that command and look at the returned properties and see an error, that means you do not have the correct permissions to move forward. You can check your permissions in the Power Platform admin center: https://admin.powerplatform.microsoft.com/

/end of update

Think about it: someone in your company creates a Flow that runs when a SharePoint item is updated. Fast forward a year or so, and that coworker has moved on, and the Flow needs to be updated. If you work for a small company or one that hasn’t fallen in love with Power Platform and Flow, you’re likely in luck, and finding the Flow will take a few minutes. In my case, there are currently 2,712 Flows in my tenant that span several environments.

The PowerShell script I’ve created will query a tenant using the Get-AdminFlow command, return all Flows, and then loop through them. The script can be adjusted to target a single environment using the EnvironmentName parameter. Note: running the script using the Get-Flow action will return all the Flows your AD account can access.

#Install-Module AzureAD
#Install-Module -Name Microsoft.PowerApps.Administration.PowerShell  
#Install-Module -Name Microsoft.PowerApps.PowerShell -AllowClobber 

#connect-AzureAD

function Get-UserFromId($id) {
    try {
        $usr = Get-AzureADUser -ObjectId $id
        return $usr.displayName
    }
    catch {
        return $null
    }
}

#get all flows in the tenant
$adminFlows = Get-AdminFlow 

#set path for output
$Path = "$([Environment]::GetFolderPath('Desktop'))\Flow_Search_for_SharePoint_$(Get-Date -Format "yyyyMMdd_HHmmss").csv"

#set target site
$targetSPSite = "https://yourTenant.sharepoint.com/sites/yourSITE"
$targetSPList = "4f4604d2-fa8f-4bae-850f-4908b4708b07"
$targetSites = @()

foreach ($gFlow in $adminFlows) {

    #check if the flow references the target site
    $refResources = $gFlow.Internal.properties.referencedResources | Where-Object { $_.resource.site -eq $targetSPSite }

    #check if the flow references the target list
    #$refResources = $gFlow.Internal.properties.referencedResources | Where-Object { $_.resource.list -eq $targetSPList }

    if ($refResources -ne $null) {

        #optional - get the user who created the Flow
        $createdBy = Get-UserFromId($gFlow.internal.properties.creator.userId)

        $row = @{}
        $row.Add("EnvironmentName", $gFlow.EnvironmentName)
        $row.Add("Name", $gFlow.DisplayName)
        $row.Add("FlowEnabled", $gFlow.Enabled)
        $row.Add("FlowGUID", $gFlow.FlowName)
        $row.Add("CreatedByUser", $createdBy)
        $row.Add("CreatedDate", $gFlow.CreatedTime)
        $row.Add("LastModifiedDate", $gFlow.lastModifiedTime)
        
        $targetSites += $(new-object psobject -Property $row)
    }
}

#output to csv
$targetSites | Export-Csv -Path $Path -NoTypeInformation

If you don’t want to get the display name of the user who created the Flow, comment out the part of the script that calls the Get-UserFromId function, and you won’t need to connect to Azure.

And to answer my original question: How do you find all the Flows running on or referencing a SharePoint list?
In the script, comment out the part of the script that references $targetSPSite and un-comment $targetSPList. You can get the GUID of the list by navigating to list settings and looking at the URL. Another option is to open the list, view the Page Source, then look for the “listId” property.

In a future post(s), I will outline how to search for all Flows that use different connectors, Dynamics 365 tables (dataverse), triggered from Power Apps, or other objects. All of the info is in the properties of the Flow; getting to it can be a little fun.

Power App and SharePoint List Form Hide Field on New Item Form

How do you hide a field on a PowerApp when opening a new form? The approach below uses a single screen form instead of multiple screens for the various forms.

I started by creating a new SharePoint list and added two text fields:
Not on New Form
On New Form
Using the customize form option, I entered the Power App designer.

When the PowerApp designer opens, it will look like this:

To help see what’s going on with the form mode, add a text label to the form and set its Text property to: "Form Mode: " & Text(SharePointForm1.Mode)

Select the field (Data Card) that should not appear on the new item form, then select the Visible property. For the Visible property, enter the following: If(SharePointForm1.Mode = 1, false, true) . If your SharePointForm1 is named something else, use it instead of the value I presented.

Breaking down the formula a little: If the SharePoint form mode is equal to 1, visible should be false, else true.

Save and publish the app, then check if it’s functional as planned.

New item form with Form Mode: 1

Display item form with Form Mode: 2

Edit item form with Form Mode: 0

Azure Runbook Job Name error: Token request failed..Exception

When you move from a SharePoint on-prem environment to SharePoint Online, you lose the server-side environment you’d normally use to run PowerShell scripts or tasks to interact with SharePoint. In my opinion, and please correct me if I’m wrong, the closest thing to a server-side environment in a cloud environment is Azure Runbooks or Azure Function Apps. I went with Azure Runbooks due to its ability to handle long-running tasks.

The error I recently encountered in my runbook was: runbook name error: Token request failed..Exception . At first, I thought there might be something wrong with the way I was connecting to Keyvault, but that wasn’t it. Next was my connection to SharePoint, this is handled using a SharePoint-generated client ID and secret. Oddly enough, I had just updated this a few months back, so it wasn’t an obvious candidate for a failure point.

I went to my target SharePoint site, created a new set of credentials using siteName/_layouts/15/AppRegNew.aspx and siteName/_layouts/15/appinv.aspx. After creating the credentials, I went back to the runbook and plugged them in, and it worked!

Long story short, if you get this error: Token request failed..Exception try creating a new client ID and secret and see if it helps clear things up.

You can also use this script to test your client id and secret. Connect-PnPOnline | PnP PowerShell

$siteUrl = "https://taco.sharepointonline/sites/burrito"
$testConn = Connect-PnPOnline -Url $siteUrl -AppId "1111-2222-3333-4444-555555555555" -AppSecret "X3tssvCebdl/c/gvXsTACOajvBurrito=" -ReturnConnection
$list = Get-PnPList "Tacos"
Write-Output $list

Use Python to Query a LARGE SharePoint list.

When querying a SharePoint list that has more than 5,000 items, you’ll likely receive an error like this:

This view cannot be displayed because it exceeds the list view threshold (5000 items) enforced by the administrator. 


Microsoft.SharePoint.SPQueryThrottledException', 'The attempted operation is prohibited because it exceeds the list view threshold.', "500 Server Error: Internal Server Error for url

Or, your query will only return the default 100 items. To get around this, pagination can be used to query the list and return ALL of the items.
Example:
all_items = list_to_export.items.paged(1000).get().execute_query()

Full script using VS Code:

from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext


app_settings = {
    'url': 'https://taco.sharepoint.com/sites/queso/',
    'client_id': 'ID here',
    'client_secret': 'shhhh its a secret',
}

context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'], client_secret=app_settings['client_secret'])

#connect to the site
ctx = ClientContext(app_settings['url'], context_auth)
ctx.execute_query()

#get the target list
list_title = "List of Tacos"
list_to_export = ctx.web.lists.get_by_title(list_title)

#get all of the list items
all_items = list_to_export.items.paged(1000).get().execute_query()
list_items = [item for item in all_items]

print("Item count: {0}".format(len(list_items)))

The example above connects to a SharePoint site using a client ID and secret, then queries the list. Again, the key here is using pagination (paged). You can adjust the page size to better fit your needs, but be sure to leave it under 5,000, or you will be back to square one.