Flow and SharePoint Online Large Libraries

Using Flow to get or check for files in a large SharePoint library can be a little tricky. If you are sure your library will always stay under 5,000 items the Get Files (properties only) Flow action is a quick n’ simple approach to use. When your library crosses over the mythical 5k mark or somewhere in that neighborhood, the Get Files action will fail to return results without warning. What I’m outlining below are other options when working with large libraries.

Setup:



Option 1: Get Files using an Odata filter query
downside: only use this in small libraries



Result:


Option 2: use the SharePoint API
downside: the lack of transparency from Microsoft related to how often data is crawled.

Result:

Option 3: use the SharePoint API along with a filter action on the library. This option does require that you have metadata set up on the library to filter on and there is not a wildcard / contains option. The filtered value needs to be exact.
downside: you will need to set up your metadata ahead of time or create it after the fact then backfill.

Result:

The more I learn about Flow and SharePoint Online, the more I’m starting to like Option 3 when doing a lookup type of search. SharePoint Search is an extremely powerful tool if the search index is fresh.

Download a File From SharePoint Online Using Python

How do you download a file from a SharePoint Online library using Python?

Items needed to run the script in this example:
Install and run a simple Python script.
Office365 Rest Python Client library:
https://pypi.org/project/Office365-REST-Python-Client/
SharePoint App Only Client Id and Secret:
Microsoft documentation:
https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs
You can create an app principle that is limited to a single site, list, library, or a combination of them:
https://piyushksingh.com/2018/12/26/register-app-in-sharepoint/

from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext
from office365.sharepoint.files.file import File

app_settings = {
    'url': 'https://YOURtenant.sharepoint.com/sites/somesite/',
    'client_id': '12344-abcd-efgh-1234-1a2d12a21a2121a',
    'client_secret': 'Oamytacohungry234343224534543=',
}

context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'], client_secret=app_settings['client_secret'])

ctx = ClientContext(app_settings['url'], context_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()

response = File.open_binary(ctx, "/Shared Documents/Invoice.pdf")
with open("./Invoice.pdf", "wb") as local_file:
    local_file.write(response.content)

If the above script does not work, take a step back and make sure you are connected to the site.

from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.sharepoint.client_context import ClientContext
from office365.sharepoint.files.file import File

app_settings = {
    'url': 'https://YOURtenant.sharepoint.com/sites/somesite/',
    'client_id': '12344-abcd-efgh-1234-1a2d12a21a2121a',
    'client_secret': 'Oamytacohungry234343224534543=',
}

context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'], client_secret=app_settings['client_secret'])

ctx = ClientContext(app_settings['url'], context_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()

print("Site title: {0}".format(web.properties['Title']))

Use Flow to Get Files Created or Modified Today in a SharePoint Library

Scenario:
Each day I have a couple of Azure Runbooks export SharePoint list items and upload them to a SharePoint library. If one of the Runbooks fails, I needed to send an email alert that something went wrong.

Basic logic:
If files created today in SharePoint <> X, send an email.

The easy solution would have been to loop through the files, check their created date, increment a variable, then make a condition statement.

More-better way:
Run flow daily at 6:00 PM
Send an HTTP request to SharePoint to get files
Parse the response
Condition statement
— if true, send an email

Uri text from the HTTP call:

_api/search/query?querytext='Path%3Ahttps%3A%2F%2Fsharepointed.sharepoint.com%2Fsites%2Fsitename%2Fsubsite%2Fexports%2F*%20LastModifiedTime%3Dtoday'

Parse JSON schema

{
    "type": "object",
    "properties": {
        "odata.metadata": {
            "type": "string"
        },
        "ElapsedTime": {
            "type": "integer"
        },
        "PrimaryQueryResult": {
            "type": "object",
            "properties": {
                "RelevantResults": {
                    "type": "object",
                    "properties": {
                        "TotalRows": {
                            "type": "integer"
                        },
                        "TotalRowsIncludingDuplicates": {
                            "type": "integer"
                        }
                    }
                }
            }
        }
    }
}

Use PowerShell PNP to Create an Alphabetical Directory of Folders in SharePoint

I’m in the process of reorganizing a document library and wanted to store all of the documents in alphabetical folders. Yes, I’m using metadata, but I’ve passed the magic 5,000 item threshold and want to rearrange the library and leverage a rich search experience.

So, using PowerShell, how do you create a bunch of folders going from A to Z?

$siteURL = "https://sharepointed.sharepoint.com/sites/parent/child"

$conn = Connect-PnPOnline -Url $siteURL -Credentials (Get-Credential) -ReturnConnection

try{
(65..(65+25)).ForEach({     
$xy = [char]$_    
Add-PnPFolder -Name $xy -Folder "/mylibrary" -Connection $conn
})
}

catch{ Write-host -f Red "Error:" $_.Exception.Message}

More information about creating folders using ASCII:
https://devblogs.microsoft.com/scripting/use-powershell-and-ascii-to-create-folders-with-letters/

Get Files From a Folder Using PNP

How do you get all of the files from a folder in SharePoint using PowerShell PNP?

  $devConn = Connect-PnPOnline -Url "https://sharepointed.sharepoint.com/sites/siteA/siteB" -Credentials $userCredential -ReturnConnection

    $folderName = "/Shared Documents/myfolder/anotherfolder"

    $folderItems = Get-PnPFolderItem -FolderSiteRelativeUrl $folderName -Connection $devConn

    foreach($item in $folderItems)
    {
        Write-Host $item.Name
    }

    Write-Host "done"

You could also use a search query with a path filter to get the files, just depends on your needs.

Example of using the Get-PnPListItem cmdlet with the FolderServerRelativeUrl parameter.

$devConn = Connect-PnPOnline -Url "https://sharepointed.sharepoint.com/sites/siteA/siteB" -Credentials $userCredential -ReturnConnection

    $folderName = "/sites/spdev2/bw2/Shared Documents/myfolder/anotherfolder"

    $folderItems = Get-PnPListItem -List "Shared Documents" -FolderServerRelativeUrl $folderName -Connection $devConn 

    foreach($item in $folderItems)
    {
        Write-Host $item
    }
    

Flow Trigger On SharePoint Item Version

How do you run a Flow on a specific version SharePoint item version?

Create a Flow, then navigate into the Settings of the first step. Scroll down to Trigger Conditions and enter the following:

@equals(float(triggerBody()?['{VersionNumber}']),1.0)

Save the Flow and run a test.
The Flow should only process items / documents where the version is equal to 1.0.

Flow Variables Not Displaying In The Dynamic Content Menu

I was trying to dynamically set a SharePoint list item ID to a variable but the variable was not displaying in the Dynamic Content menu.

The solution was to use an expression to set the value, save the flow, exit the edit screen, then re-enter the edit screen.

variables('varRefID')
Create a variable
Notice the variable is not available for selection
Use the expression window to enter the variable name
The variable has been added but looks weird. This is not an issue but continue reading if you want to clean display value.
Save your flow and exit the edit screen.
Edit your flow
The variable is now displayed correctly!

Copy Files From Azure File Storage to SharePoint

As of today, there is not a Logic App trigger for Azure File Storage, so I went with a schedule-based approach. Yes, this example leaves out a lot of fine-tuning, but it will get you headed in the right direction.

Create a blank Logic app
Trigger: Schedule
Action: Azure File Storage – List files
Action: SharePoint – Create file

After you add the SharePoint action, the Logic App should automatically add a For Each action and place the SharePoint Create File action inside of it.

Overview of the Logic App
For each action expanded
Testing the Logic App

In the last screenshot, I tested the Logic App by uploading a couple of documents in Azure Storage Explorer, then I manually ran the Logic App (click the Run button).

Again, this is a simple example. The example does not account for processing the same files over and over…

SharePoint Search Query Tool Login

If you have ever worked with SharePoint search you likely already know about the SharePoint Search Query Tool. If you are new to SharePoint and need a little insight into the SharePoint search experience this tool is a lifesaver!

SharePoint Query Tool GitHub: https://github.com/pnp/PnP-Tools/

In future posts, I will outline how to form queries and use the tool but for now, I want to simply connect to my SharePoint Online site.

Enter the URL for your SharePoint site, select the Authentication options shown above, then click Sign-In. If a web login form appears be sure to complete it. If your normal Windows login doesn’t work, try using your work email address and password, and if that doesn’t work try your work email address and App Password.

App Passwords are created and managed at this URL: https://account.activedirectory.windowsazure.com/AppPasswords.aspx